{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#
Information Retrival and related technologies
\n##
Topic: Text Data
\n###
By Preethi, Sakshi and Satiya
","metadata":{}},{"cell_type":"markdown","source":"## INTRODUCTION\n\n\"AI will be the defining technology of our time, just as the steam engine and electricity were for the Industrial Revolution.\" - Satya Nadella, CEO of Microsoft\n\n> Artificial Intelligence(AI) has become entwined with every facet of our lives, captivating the imaginations of scientists and researchers. It might seem like it is a recent development but, in reality, the groundwork started all the way back in the 1900s, with the progress made in the mid-20th century, when Alan Turing, a mathematician and computer scientist, proposed the question, \"Can machines think?\" and ever since there have been colossal advancements and breakthroughs, where it has been developed from a mere concept into becoming a potent tool, transforming the way we live, and work and reshaping the way to perceive and interact with technology. \n \n> Languages have been a fundamental medium for human communication which could be in the form of text or speech. While humans can understand phrases such as “Blah”, “Meh”, “Ouch” etc effortlessly, we can't help but wonder if machines could comprehend such nuances. This disparity highlights the ongoing need for more developments to be made to bridge the gap between human language and artificial intelligence (AI). While AI has made remarkable improvements in language processing tasks such as translation, text generation etc, they still need development to capture subtleties and intricacies embedded in human communication. To bridge this gap, developers have been continuously striving to improve Natural Language Processing techniques and models, to enhance the ability of the systems to understand the meaning and interpret them accurately. To accomplish this, many datasets, algorithms and models have been leveraged. \nNow that we know that IR systems are responsible for providing relevant information as per the interest of the user, they are also expected to provide results that are recent to the good quality data and the systems must cover a wide range of domains. It is also crucial that the retrieval is completed in minimum time. \n\n# WORKING OF IR\n\n> An IR system operates by the collection of documents, such as web pages, articles etc, this is carried out by a component, namely the crawler or the Spider. The system creates an index using the documents collected, by making use of an indexer, which analyzes the documents, extracts relevant terms and maps terms to the documents in which they appear. When the user inputs a query, the query is processed and matched to the indexed documents, this process is carried out by Query Processor. The Ranking Algorithm determines the relevance of the documents to the given query and ranks them. These documents are then presented to the users. \n\n## INFORMATION RETRIEVAL AND RELATED TECHNOLOGIES\n\n> Information Retrieval(IR) is a key technology which deals with the process of searching, obtaining and processing information the user is in need of, from a large knowledge base. This is done in the form of queries, where the user requests the required information in the form of a query. The representation of the documents is then compared to the query by using a similarity function and the relevant documents are extracted. IR involves the use of multiple specialized techniques, algorithms and models for the retrieval of information. The further evolution of NLP, deep learning and machine learning led to the emergence of Recommendation systems and Large Language Models(LLMs), tools that help people be more productive and efficient. LLMs such as GPT-3, BERT, etc are models that are trained using deep learning techniques on enormous data, which are capable of performing NLP tasks, while recommendation systems are tools that suggest relevant suggestions based on the user's preferences. While they still have many limitations, these systems demonstrate promising in information retrieval, content generation and enhancing user experience. \n\n## RECOMMENDER SYSTEMS\n\n> A subclass of machine learning predicts rating the data according to the requirements specified by the user, which are then ranked and returned to the user. They are used by various companies such as Netflix, Spotify, Google etc, to increase user engagement with the platform. The quality of these recommendations made can be assessed by measuring the accuracy and coverage. Accuracy is the ratio of correct recommendations made by of the total possible recommendations and coverage is the measure of how well the system can suggest items from the entire search space. There are many ways to evaluate the accuracy of a recommendation system such as RMSD, MAE, K fold validation etc. \n\n> Much of today’s Recommender Systems use Neural Networks, where hidden or new patterns are not discovered. Hence, [1] proposes the use of Epistemic Neural Recommendation (ENR), a novel architecture based on neural contextual bandit algorithm and Thompson sampling as an exploration strategy. The authors argue that this method improves the personalization and exploration capabilities of the model. The recommendation system interacts with the environment by making recommendations based on user preferences and receives rewards for its actions. Thompson sampling is an algorithm that samples reward estimates from context-action pairs' posterior distributions and selects actions greedily based on these samples. ENR incorporates the estimation of epistemic uncertainty, which allows the agent to understand what it does not know and guides exploration. Epistemic uncertainty estimation is supported by a scalable approach called EpiNet. This experiment aims at improving the click-through rates and user rating, moreover with lesser interactions from the user and with much lesser need for computational resources, compared to the other neural network algorithms. Furthermore, the model requires 20% lesser user interactions to accomplish the result of a better baseline algorithm. \n\n> While [1] explains the importance of finding the hidden patterns in the data, [4] focuses on the selection of the appropriate baseline from the number of proposed and already existing baselines. This paper makes use of RecBaselines2023, a dataset which consists of about 900 papers with a total of 363 baselines, which were proposed between 2010 and 2022. The method used to pick the best algorithm is the Collaborative Filtering Technique, which is a method where the model makes predictions by making identifying the interest and information of other users. The general idea is that if person X and person Y have the same choice in a product, it is likely that they have a similar taste in other products as well. This model is simple to implement and provides high coverage. This recommendation is thereon compared to sever other state-of-art top-N recommendations models using the same dataset. This paper also discusses the number of papers that were published in this area of research over the past years and collects the obtain papers that are to be used in the dataset. This data of papers are then preprocessed to arrive at 900 papers with a total of 363 baselines from 1009 papers with 2187 baselines. This dataset includes interactions between the paper and the baselines and also, the new dataset that has been introduced can help researchers to compile baselines for their work in the future. \n\n> The research suggests a unique method for multimedia recommendation named MODEST (MOdalityDEcorrelation for STable learning). By estimating a weight for each item, the proposed technique tries to decorrelate the properties of the weighted distribution that come from various modalities. The Hilbert Schmidt Independence Criterion (HSIC), a kernel-based method that may assess the degree of correlation between two multidimensional and non-linear variables, is used by the algorithm as an independence testing measure. In the training phase, the method alternately optimises sample weights by minimising modal correlation in the weighted distribution (measured through HSIC loss) and model parameters by minimising weighted Bayesian Personalised Ranking (BPR) loss. The trained multimedia recommendation models can be utilised directly to generate suggestions during the inference phase. By integrating MODEST into four cutting-edge models and doing experiments on four types of widely-used Amazon review datasets, the authors assess MODEST's efficacy. By decorrelating various modalities and directing the backbone models to concentrate on the modalities that can reveal users' true preferences instead of irrelevant modalities, the proposed method is well-designed for multimedia recommendation, as evidenced by the results showing that MODEST outperforms the baseline models in terms of Recall@20. The suggested method can be put to use in a variety of sectors, including e-commerce, advertising, and social media, where user engagement and revenue are heavily dependent on the recommendation of multimedia material.\n\n> Another such paper file introduces the problem of multi-distribution dense information retrieval and includes benchmarks and evaluations for this task. The authors suggest a method for extracting two passages, one from each corpus, that are most pertinent to a given query. They define two sets of text passages with different features, taken from different corpora. The authors demonstrate that their techniques are consistent when fine-tuning various basic retrieval models and result in notable recall gains across datasets. The authors recommend that future work can concentrate on developing larger datasets specifically suitable for this task. They acknowledge the shortcomings of their work, notably the relatively modest size of some datasets. Overall, the suggested techniques are openly accessible and adaptable to different retrieval models.\n\n>A unified framework for the extraction of multimodal features in recommendations is presented in the study \"Ducho: A Unified Framework for the Extraction of Multimodal Features in Recommendation\". The framework can access shared interfaces to extract information from several multimodal recommendation systems while effectively and equitably comparing them. The paper examines the advantages of utilising Ducho for multimodal recommendation, including its usage of a YAML-based file setup and its capacity to incorporate several deep learning libraries as backends. The construction of the Ducho pipeline, which entails preprocessing, loading, creating models, extracting features, and producing results, is also covered in the study. The Dataset, Extractor, and Runner modules work together to manage the pipeline. The possible uses of Ducho in a variety of industries, such as fashion, music, cuisine, and micro-video suggestion, are covered in the paper's conclusion. The research offers a thorough methodology for the extraction of multimodal information in recommendations, which could enhance the effectiveness of recommendation algorithms across a range of disciplines. Transformers, PyTorch, and TensorFlow are some of the AI models employed in the framework.\n\n## LARGE LANGUAGE MODELS \n\n> Large Language Models, which are based on deep learning architectures, synthesize enormous amounts of text data and trained them to understand them and to generate information relevant to the original content. LLMs are trained on a corpus of data that is publicly available, the model then learns to predict and capture the patterns in the data. After the pre-training process, the LLMs are fine-tuned on the datasets to make them provide specific solutions, this helps the model adapt to the target and enhance the performance. The information captured during the pre-training is used to generate results according to the user's preferences. These models are then iteratively refined to keep up with new language patterns and emerging information. While LLMs seem to look to Search Engines, LLMs possess the capability to understand and interpret the queries, which results in more precise and appropriate replies according to the behaviour of the user and the information provided.\n\n
\n\n
\n\n> OpenAI’s ChatGPT-3, a model built and trained on colossal amount of data from the internet, gives it the understanding of languages and knowledge on various myraids of topics. Although these capabilities seem impressive, sometimes they confidently generate incorrect answers that might appear convincing, hence it is necessary to verify these answers before accepting them as accurate. In [5], we consider if it is possible to detect LLM hallucinations with the help of an information retrieval system to retrieve supporting evidence, which is then checked against generated responses by the LLM itself. The author experiments to explore if the LLMs can self-detect hallucinations by confirming them against external corpus. And finds that with the help of information retrieval, it can identify its own hallucinations to an extent. The question is given to the LLM, which is then prompted for an answers. The generated answer is then combined with the original question to form a query, which is used to find the appropriate passage from the corpus, which will be called the retrieved answer. The question, generated answer and the retrieved answer is now sent into the LLM to determine if the generated and retrieved solutions are consistent. Output of which could be of 3 types: \n> “YES” - No hallucination involved, the solutions are the same.\n> “NO” - There is hallucination involved, the solutions are not the same.\n> “NOT RELATED” - Both solutions are not related to the questions. For example, the LLM could give us a reply saying it needs more information to find the required solutions.\n\n> This type of studies help us enhance the reliability of AI generated solutions by detecting and addressing the hallucination, helps improve the accuracy and reliability of the generated answers. \n\n> While [5] helps us understand the possible ways in which we could work on developing Large Language System, [6] becomes a case study where ChatGPT GPT-3 was used for Product Information Extration. This paper focussed on the sage of ChatGPT to extract Metadata of products, for e-commerce, from title and description of the product, in addition it focussed on\n> Evaluating the performance in comparison to pre-trained models such as BERT.\n> Evaluating the performance for open and closed extraction\n> Exploring different prompt design\n> Evaluating the in-context learning for ChatGPT\n> This paper uses a subset of the MAVE dataset to test the performance of ChatGPT. The researchers use a subset of this dataset and perform the experiment, then compare the results to that of the baseline models such as AVEQA and NER(named entity recognition model), which are fully and extensively trained models that need extensive computational resources. This paper combines demonstrations with prompts to improve the product information extraction and uses precision, recall and F1 score. \n\n> ChatGPT is given demonstrations(sample inputs and outputs which helps the model in learning), after providing task description, which provides an overview of the task. This helps in the learning of tasks for the model. After which the task description is passed, followed by the actual input for the model to give the appropriate response based on the learning. \n\n> The OpenAI text-ada-002 model is notably mentioned in the paper's discussion of embedding models' application to natural language processing. On a held-out test dataset, the authors trained a neural network to transform open-source MPNet embeddings into text-ada-002 embeddings, averaging an average cosine similarity of 0.932. A subset of 50,000 reviews from a dataset of 568,454 reviews of gourmet foods on Amazon were chosen for their research. The authors also go over their process for leveraging the text-ada-002 and all-mpnet-base-v2 models to obtain vector embeddings for the reviews. They used the LightspeedEmbeddings package to enable multithreading, split lengthy reviews into 128-word chunks, compute average embeddings, and submit many API queries at once. The quality of the authors' anticipated \"synthetic\" embeddings for vector search on text-ada-002-embedded data was manually evaluated.\n\n> In order to analyze the themes that were popular on Twitter during the COVID-19 epidemic and comprehend the crucial elements that changed the public's perception of vaccination, this article offers an interactive visualization tool. The authors draw attention to how difficult it was for healthcare organizations to meet their targets for vaccine coverage due to the pandemic's unprecedented dissemination of false information and anti-vaccine propaganda. The suggested solution is simply generalizable to any scenario for visual analysis and to promote social media data transparency for both scholars and the universal public. The authors depict the cumulative position over time using bubble charts, where each bubble represents a tweet and the colour of the bubble distinguishes the individuals. The study places a strong emphasis on the value of comprehending the information spreading via social media and taking an active part in (de)motivating the general population to get immunised. The authors contend that persuading people to get immunized depends on their ability to comprehend the facts giving rise to their worries. The potential of the suggested tool to improve social media data transparency and understanding for researchers and the general public is highlighted in the paper's conclusion.\n> \n
\n\n
\n\n## QUESTION ANSWERING SYSTEMS\n> \tQuestion Answering has been a long-standing and critical problem in the field of Natural language Processing and Artificial Intelligence, is considered to be a critical milestone in the recent times. These systems enable users to express questions in Natural Language and get appropriate results immediately. The typical Question Answering Sytems contains the following: \n* > Data source,\n* > Information retrieval (IR) system,\n* > Machine reading comprehension model (MR model),\n* > One or more additional blocks, such as modules for text pre-processing, answers post processing, checking, and stabilization.\n\n
\n\n
\n\n> When a question is put forward, the system first searches the answer to the question from the text, the sources which could be web pages, documents, Wikipedia etc. It, then, retrieves the most appropriate imformation front the Data Source, for which many algorithms can be employed. \n\n> The Standford Question Anwering Dataset(SQuAD) was a widely used dataset in training and evaluating models to comprehend the passages of text and answrig the questions based on the passages. A paragraph from the articles that are used as data sources are called PASSAGES, and they can be of any length. The SQuAD was made using a collection of Wikipedia pages that covered myriads of domains. The dataset consists of set of questions from each passage, to which the answers to these questions can found in the respective passage. One noteworthy fact about the dataset is that, the answers are not limited to single-word responses or named entities and allows more diverse and complex answers. \n\n> QA systems face numerous challenges, such as understanding complex questions, context and providing accurate and reliable answers and various techniques and methods have been employed to address these challenges. The goals of these systems is to improve user experience by providing relevant and accurate answers to the questions put forth by the users, saving time. The continued improvements in the field of NLP, machine learning, and large-scale datasets like SQuAD have contributed to the development and improvement of QA systems make them more robust and efficient systems. \n\n
\n\n
\n\n> The traditional QA systems rely on only one source of knowledge, which often limits the capabilities of the system from fairly evaluating the answers. Hence, to overcome this limitation, [4] proposes the use of heterogenous sources, which improves the coverage and confidence by harnessing solutions from multiple sources such as databases, websites, documents etc for answering Factual questions. While numerous have been done using the several existing datasets, they were primarily design with the assumption that there exists a specific underlying source that contains nearly all the answers to the questions, which limits the usefulness of the QA systems. \n\n> To analyse and assess the performance of the heterogeneous QA, the authors have designed a new Benchmarks called CompMix which consists of 9410 questions covering 5 different domains, split into train set (4,966), development set (1,680), and test set (2,764). By introducing CompMix, the authors aim to provide a more suitable testbed for evaluating the performance of heterogeneous QA systems in realistic scenarios where multiple sources are needed to obtain accurate answers. \n> (i) it is crowdsourced;\n> (ii) it includes the full KB as one of the knowledge sources; \n> (iii) it spans four sources for accurate answers\n> (iv) it covers five domains; and \n> (v) it contains self-contained complete questions.\n> Answers are from Wikidata which enables consistent evaluations. \n\n> While [4] focusses more Fact-centric Question Answering Systems, [10] focusses on community Personalised Question Answering where the paper introduces a new dataset - SE-PQA(StackExchange - Personalized Question Answering). The paper also highlights the advantage of combining multiple community data for personalisation. Personalised search aims at giving results that are tailored for an individual user’s interest and behaviour. This paper highlights the use of Deep Neural Networks to train the personalised models and also that there are not public-available, high-quality dataset for the same, which the existing ones have many limitations. The dataset is based on the popular cQA platform, StackExchange, with 1 million questions and 2 million answers, which captures social media interactions such as votes received by questions and answers, topic description using tags etc. \n\n> The relevance of an answer to a question is inferred based on the number of up-votes given by the community members. The experiments involving personalised cQA models consider only the answer explicitly labelled as the best answer by the user who asked the question as relevant.\n> Cleansing - Discarding the questions with no answers and the answers with negative scores (Low quality and not relevant to the cQA). The answers selected as the best answer by the user are considered.\n> Two versions of the dataset: the base version and the personalised version. In the base version, all answers with positive scores are considered relevant for a question. In the personalised version, only the single answer selected as the best by the user is considered relevant for both the user and the question. \n> Personalisation Phase - Tailoring the questions and answers to better suit the user’s preferences that are targeted and useful. Increasing the user experience, accuracy and effectiveness of the system by considering the user-specific data. Various user-generated information are used, such as previous questions and answers, social interactions with other users, tags assigned to previous questions, earned badges, biographic information, and numeric features (reputation score and voting history). This helps the system identify patterns, interests and areas of expertise. \n\n> Two-Stage Ranking System - Balancing the effectiveness and efficiency in finding the most relevant document for a Query.\n> Recall (Retrieving a large set of candidate documents that are potentially relevant) with a ElasticSearch(Search engine) and Fast ranking algorithm - BM25.\n> Precision (Selecting the most accurate and relevant documents from the candidate set identified in the first stage) - Re-ranks the documents computed in stage 1,using MiniLMSBERT and DistilBERT are used\n\n> The personalised model called TAG incorporates user interests and tags associated with questions and answers to compute a personalization score for each answer, an answer is assigned a higher score if the user who asked the question shares similar interests with the user who provided the answer. The final ranking is obtained by the score from individual models (BM25, Neural, and TAG) are combined using weighted sums. The normalized scores from each model are multiplied by their respective weights and then summed together. \n\n## CONCLUSION\n> In conclusion, the field of information retrieval continues to advance, with researchers exploring various avenues to improve recommendation accuracy, handle long-term user preferences, address ethical concerns, and maintain up-to-date datasets. The future of information retrieval lies in incorporating more contextual information, leveraging advanced neural network architectures, and experimenting with different training methodologies. Additionally, the field aims to tackle bias and fairness issues, ensure accurate extraction of information, and develop more sophisticated models for personalization. Furthermore, there is a need to expand research beyond NER and AVEQA, explore other domains, and integrate data from multiple sources. NLP techniques play a crucial role in enhancing language understanding, machine translation, and performance optimization in information retrieval systems. Future work should focus on robustness, interpretability, multilingual understanding, domain-specific adaptation, and lifelong learning. By addressing these challenges and exploring new possibilities, the field of information retrieval can continue to make significant advancements in meeting the diverse needs of users in the digital age. \n\n## REFERENCES \n1. Scalable Neural Contextual Bandit for Recommender Systems, 2023, https://arxiv.org/pdf/2306.14834 \n1. “What Is Artificial Intelligence (AI)?”, n.d., https://cloud.google.com/learn/what-is-artificial-intelligence \n1. Revolutionizing Information Retrieval: The Role of Large Language Models in a Post-Search Engine Era, 2023, https://medium.com/@daniele.nanni/revolutionizing-information-retrieval-the-role-of-large-language-models-in-a-post-search-engine-7dd370bdb62 \n1. CompMix: A Benchmark for Heterogeneous Question Answering, 2023, https://arxiv.org/pdf/2306.12235 \n1. Retrieving Supporting Evidence for LLMs Generated Answers, 2023, https://arxiv.org/pdf/2306.13781.pdf \n1. Product Information Extraction using ChatGPT, 2023, https://arxiv.org/abs/2306.14921 \n1. End to End Question-Answering System Using NLP and SQuAD Dataset, 2022, https://www.analyticsvidhya.com/blog/2021/11/end-to-end-question-answering-system-using-nlp-and-squad-dataset/ \n1. Automatic Question Answering, 2018, https://towardsdatascience.com/automatic-question-answering-ac7593432842 \n1. RecBaselines2023: a new dataset for choosing baselines for recommender models, 2023, https://arxiv.org/pdf/2306.14292 \n1. SE-PQA: Personalized Community Question Answering, 2023, https://arxiv.org/pdf/2306.16261 \n1. Visualising Relation Between (De)Motivating Topics and Public Stance toward COVID-19 Vaccine https://arxiv.org/pdf/2306.12118.pdf \n1. Fuzzification-based Feature Selection for Enhanced Website Content Encryption https://arxiv.org/ftp/arxiv/papers/2306/2306.13548.pdf\n1. Ducho: A Unified Framework for the Extraction of Multimodal Features in Recommendation https://arxiv.org/pdf/2306.17125.pdf\n1. Exploring the Representation Power of SPLADE Models https://arxiv.org/pdf/2306.16680.pdf\n1. Mining Stable Preferences: Adaptive Modality Decorrelation for Multimedia Recommendation https://arxiv.org/pdf/2306.14179.pdf\n1. Resources and Evaluations for Multi-Distribution Dense Information Retrieval https://arxiv.org/pdf/2306.12601.pdf \n1. A Compact Neural Network Approach for Transforming Text Embeddings with High Fidelity https://arxiv.org/ftp/arxiv/papers/2306/2306.12689.pdf \n1. Efficient Partitioning Method of Large-Scale Public Safety Spatio-Temporal Data based on Information Loss Constraints https://arxiv.org/ftp/arxiv/papers/2306/2306.12857.pdf ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}}]}