{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.display import Image\nImage(\"/kaggle/input/tab-pic/Cover.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.573696Z","iopub.execute_input":"2023-07-04T16:23:09.574168Z","iopub.status.idle":"2023-07-04T16:23:09.673921Z","shell.execute_reply.started":"2023-07-04T16:23:09.574136Z","shell.execute_reply":"2023-07-04T16:23:09.672972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a href=\"https://www.freepik.com/free-vector/blue-3d-particles-background-design_3086334.htm#query=artificial%20intelligence%20background&position=27&from_view=search&track=ais\">Image by starline</a> on Freepik","metadata":{}},{"cell_type":"markdown","source":"# Tabular Data in the Age of AI\n---\n## Key findings\n\n- Tabular data has a rich history, dating back to ancient civilizations, and has played a vital role in knowledge representation and decision-making.\n- Deep learning models designed specifically for tabular data have shown promise, but they still lag behind gradient-boosted tree ensembles in terms of performance and training time.\n- Self-supervised learning (SSL) has gained popularity for learning representations from unlabeled tabular data, with methods like Masked Encoding for Tabular Data (MET) outperforming existing approaches on benchmark datasets.\n- Automated machine learning (AutoML) tools, such as Autogluon, mljar, and Auto-Keras, have revolutionized the modeling process for tabular data, automating various stages and democratizing access to advanced machine learning techniques.\n- Explainable AI (XAI) plays a crucial role in building trust and confidence in machine learning models, detecting bias, and improving model performance.\n\n---","metadata":{}},{"cell_type":"markdown","source":"## 1. Introduction","metadata":{}},{"cell_type":"markdown","source":"**A**rtificial **I**ntelligence (AI) has revolutionized the way we analyze and extract insights from data. With its remarkable capabilities, AI has become an indispensable tool in the field of data science. Tabular data, in particular, holds tremendous importance as it is prevalent across a wide range of domains, including finance, healthcare, and retail, among others. The efficient analysis of tabular data enables businesses and organizations to make informed decisions, identify patterns, and uncover valuable insights.\n\nThe purpose of this report is to provide an overview of the recent advancements in AI techniques for tabular data. Our goal is to provide valuable insights to the Kaggle data science community and inspire future innovations. By grasping the latest AI techniques and their practical applications in tabular data analysis, data professionals can remain at the forefront of this rapidly evolving field.\n\n---","metadata":{}},{"cell_type":"markdown","source":"## 2. An Overview of Tabular Data","metadata":{}},{"cell_type":"markdown","source":"### 2.1 What is Tabular Data?\n\nTabular data refers to structured data that is organized in a tabular format, comprising rows and columns. Each row represents a distinct instance or observation, while each column represents a specific attribute or feature of that instance. Tabular data commonly appears in databases, spreadsheets, and comma-separated values (CSV) files. Analyzing tabular data is essential for various tasks, including decision-making, pattern recognition, and prediction.\n\nBusinesses extensively utilize tabular data to gain insights into customer behavior and support critical decisions regarding product development, pricing, and marketing strategies. Researchers also rely on tabular data to uncover patterns within scientific datasets and make predictions about future events. With its structured nature, tabular data serves as a valuable resource for data-driven analysis and informed decision-making across diverse domains.\n\n**Figure 1.** From: \"What is Tabular Data? (Definition & Example)\" by Zach Bobbit from [Statology](https://www.statology.org/tabular-data/)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Tabular.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.675776Z","iopub.execute_input":"2023-07-04T16:23:09.676247Z","iopub.status.idle":"2023-07-04T16:23:09.696135Z","shell.execute_reply.started":"2023-07-04T16:23:09.676217Z","shell.execute_reply":"2023-07-04T16:23:09.694710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A diagram of tabular data with rows that contain observations and columns as attributes.\n\n---","metadata":{}},{"cell_type":"markdown","source":"### 2.2 Types of Tabular Data\n\nTabular data encompasses a wide range of information that can be organized and represented in a tabular format consisting of rows and columns. The types of tabular data vary based on the nature of the data being represented. In this section, we explore some common types of tabular data, highlighting their characteristics and applications.\n\n**1. Numerical Data:** \nNumerical data consists of quantitative values represented as numbers. This type of data is often used in scientific and financial domains where measurements, calculations, and statistical analyses are prevalent. Numerical data can include data such as temperature readings, stock prices, sensor measurements, or experimental results. Analysis of numerical data involves statistical techniques, trend analysis, regression modeling, and other numerical computations.\n\n**2. Categorical Data:**\nCategorical data represents data that falls into distinct categories or groups. It includes variables with discrete values that do not have an inherent numerical meaning. Examples of categorical data include gender, product categories, geographic regions, or customer segments. Analysis of categorical data often involves frequency counts, cross-tabulations, and the application of techniques like chi-square tests or association rule mining.\n\n**3. Textual Data:**\nTextual data involves textual information organized in a tabular format. This type of data is commonly found in document collections, customer feedback, social media posts, or news articles. Each row typically represents a document or a piece of text, while columns may contain metadata such as timestamps, authors, or categories. Textual data analysis focuses on techniques like text mining, natural language processing (NLP), sentiment analysis, and topic modeling to extract insights from the text.\n\n**4. Temporal Data:** \nTemporal data involves time-related information, where each row represents an event or observation occurring at a specific point in time. This type of data is prevalent in domains such as finance, healthcare, and IoT. Temporal data can include timestamps, sensor readings, stock market data, or patient vital signs. Analysis of temporal data includes time series analysis, trend detection, anomaly detection, and forecasting techniques.\n\n**5. Spatial Data:** \nSpatial data incorporates geographic or spatial information into a tabular format. Each row represents a spatial entity, such as a location or region, while columns may contain attributes like coordinates, boundaries, or spatial characteristics. Spatial data is widely used in geographic information systems (GIS), urban planning, environmental studies, and transportation analysis. Analysis of spatial data includes spatial queries, spatial joins, spatial clustering, and spatial statistics.\n\n---","metadata":{}},{"cell_type":"markdown","source":"### 2.3 Historical Overview of Tabular Data\n\nTabular data has a rich history that parallels the development of data analysis and information management. Over centuries, humans have recognized the power of organizing and presenting data in tabular form, facilitating comprehension, comparison, and analysis. This historical overview traces the evolution of tabular data from ancient origins to modern-day applications, unveiling its enduring significance in knowledge representation and decision-making.","metadata":{}},{"cell_type":"markdown","source":"**1. Ancient Times - 4000 BC to 3001 BC**\n\nThe earliest examples of tabular data can be found in ancient civilizations. The ancient Egyptians carved or painted onto a variety of surfaces, including papyrus, stone, wood, and even pottery to record astronomical observations, inventories, tax records, and any other things.\n\nImage from [Mrs. Redd's Social Studies](http://www.reddsocialstudies.com/egypt-pictures-karnak-temple.html)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Wall.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.697562Z","iopub.execute_input":"2023-07-04T16:23:09.697948Z","iopub.status.idle":"2023-07-04T16:23:09.719277Z","shell.execute_reply.started":"2023-07-04T16:23:09.697916Z","shell.execute_reply":"2023-07-04T16:23:09.718416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Carvings of ancient Egyptian numbers on the walls of the Karnak Temple Complex in Luxor, Egypt. The tally mark represents 1s, the horseshoe shapes are 10s, and the coiled ropes are 100s.\n\n---","metadata":{}},{"cell_type":"markdown","source":"**2. Statistical Tables and the Enlightenment - 17th to 18th Century**\n\nThe Enlightenment era witnessed significant advancements in data collection, analysis, and representation. Pioneering statisticians like John Graunt and William Playfair introduced statistical tables and graphical visualizations in the 17th and 18th centuries. Graunt's work on mortality statistics in the [Bills of Mortality](https://www.lindahall.org/about/news/scientist-of-the-day/john-graunt) and Playfair's invention of line graphs and bar charts brought tabular data to life, enabling insights into population dynamics, economic trends, and social phenomena.\n\nImage from [Wikipedia](https://en.wikipedia.org/wiki/John_Graunt)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/John Graunt.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.720628Z","iopub.execute_input":"2023-07-04T16:23:09.721134Z","iopub.status.idle":"2023-07-04T16:23:09.736487Z","shell.execute_reply.started":"2023-07-04T16:23:09.721102Z","shell.execute_reply":"2023-07-04T16:23:09.735667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Table of Casualties from the fifth edition of John Graunt's book Natural and Political Observations Made Upon the Bills of Mortality, which was released in 1676.\n\n---","metadata":{}},{"cell_type":"markdown","source":"**3. Industrial Revolution and Data Management - 19th Century**\n\nThe Industrial Revolution brought about a surge in data generation and the need for efficient data management. The emergence of factories, railroads, and global trade necessitated systematic record-keeping and data organization. Business pioneers like Charles Babbage and Florence Nightingale recognized the value of tabular data in decision-making. Babbage's work on the Analytical Engine laid the foundation for computing machines capable of data manipulation, while Nightingale's use of polar area diagrams showcased the power of visualizing complex tabular data.\n\nImage from [Britannica](https://www.britannica.com/technology/Analytical-Engine#/media/1/22628/219565)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/AE.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.738851Z","iopub.execute_input":"2023-07-04T16:23:09.739390Z","iopub.status.idle":"2023-07-04T16:23:09.752684Z","shell.execute_reply.started":"2023-07-04T16:23:09.739358Z","shell.execute_reply":"2023-07-04T16:23:09.751552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A component of Charles Babbage's Analytical Engine (finished in 1910). This component, which was only half completed at the time of Babbage's death in 1871, features the \"mill\" (functionally equivalent to a modern computer's central processing unit) and a printing mechanism. The machine was designed with four components: a mill, a store, a reader, and a printer. Another unique feature of the Analytical Engine was the reader. Data (numbers) were to be entered onto punched cards utilizing the Jacquard loom's card-reading technology.\n\n---","metadata":{}},{"cell_type":"markdown","source":"**4. Emergence of Databases and Information Systems - Mid-20th Century**\n\nThe mid-20th century marked a significant shift in data management with the advent of computers and the development of databases. Edgar Codd introduced the relational model, which provided frameworks for organizing and querying tabular data, and IBM's IMS system proposed the hierarchical model. Relational database management systems (RDBMS) like Oracle, IBM DB2, and MySQL became fundamental tools for data storage and retrieval, enabling efficient management of large-scale tabular datasets.\n\nImage from [IBM](https://www.ibm.com/ibm/history/ibm100/us/en/icons/reldb/)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/RLDS.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.754319Z","iopub.execute_input":"2023-07-04T16:23:09.754666Z","iopub.status.idle":"2023-07-04T16:23:09.764913Z","shell.execute_reply.started":"2023-07-04T16:23:09.754640Z","shell.execute_reply":"2023-07-04T16:23:09.763971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A diagram showing how users interact with a relational database.\n\n---","metadata":{}},{"cell_type":"markdown","source":"**5. Spreadsheet Software and Desktop Computing - 1980s**\n\nThe introduction of personal computers and spreadsheet software in the 1980s revolutionized data analysis and decision support. Software like VisiCalc, Lotus 1-2-3, and Microsoft Excel empowered individuals to manipulate and analyze tabular data with ease. Spreadsheets became versatile tools for financial modeling, project management, and data exploration, driving data-driven decision-making at various levels of organizations and empowering users with interactive data analysis capabilities.\n\nImage from [WinWorld](https://winworldpc.com/product/microsoft-excel/1x)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Microsoft Excel 1.5 - About.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.766455Z","iopub.execute_input":"2023-07-04T16:23:09.767755Z","iopub.status.idle":"2023-07-04T16:23:09.778019Z","shell.execute_reply.started":"2023-07-04T16:23:09.767708Z","shell.execute_reply":"2023-07-04T16:23:09.777144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Screenshot of Microsoft Excel's early version, created exclusively for the Apple Macintosh computer. It succeeded Microsoft's previous Mutiplan spreadsheet software.\n\n---","metadata":{}},{"cell_type":"markdown","source":"**6. Big Data Era and Cloud Computing - 21st Century**\n\nThe 21st century witnessed an explosive growth in data volume, variety, and velocity, leading to the era of big data. With the proliferation of internet-connected devices, social media platforms, and online transactions, vast amounts of tabular data became available for analysis. Cloud computing technologies, such as Amazon Web Services, Microsoft Azure, and Google Cloud Platform, provide scalable storage and processing capabilities, facilitating the management and analysis of massive tabular datasets.\n\n---","metadata":{}},{"cell_type":"markdown","source":"**7. AI and Machine Learning in Tabular Data Analysis - Presently**\n\nThe integration of artificial intelligence (AI) and machine learning (ML) techniques has brought new dimensions to tabular data analysis. Sophisticated ML algorithms, including gradient boosting models, deep neural networks, and transformer architectures, have significantly advanced predictive modeling and pattern recognition in tabular data. AI-driven techniques like AutoML and self-supervised learning have automated and optimized the analysis process, enabling faster insights and empowering data-driven decision-making.\n\n---","metadata":{}},{"cell_type":"markdown","source":"## 3. Key Developments in AI for Tabular Data in the Past Two Years","metadata":{}},{"cell_type":"markdown","source":"### 3.1 The Rise of Deep Learning for Tabular Data\n\nDeep learning, traditionally used for image and text processing, has gained prominence in the analysis of tabular data. New deep learning architectures, such as DeepFM and Wide&Deep, have been specifically designed to handle tabular data.\n\n[Borisov et al. (2021)](https://arxiv.org/abs/2110.01889) conducted a comprehensive study on deep learning approaches for tabular data. They compared eleven deep learning methods with traditional machine learning approaches on popular real-world tabular datasets. Their evaluation revealed that gradient-boosted tree ensembles, like XGBoost and CatBoost, still outperformed deep learning models on most supervised learning tasks while requiring significantly less training time. However, the authors noted that research progress on competitive deep learning models for tabular data has been stagnant. They recommend future work focus on developing more efficient and effective deep learning architectures for tabular data.\n\n**Figure 2.**\nFrom: \"Deep Neural Networks and Tabular Data: A Survey\" by [Borisov et al. (2021)](https://arxiv.org/abs/2110.01889)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Figure 2.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.779640Z","iopub.execute_input":"2023-07-04T16:23:09.780365Z","iopub.status.idle":"2023-07-04T16:23:09.799326Z","shell.execute_reply.started":"2023-07-04T16:23:09.780335Z","shell.execute_reply":"2023-07-04T16:23:09.798564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Benchmarks for training (left) and inference (right) times for various methods on the Adult data set with 32.561 samples. The size of the circle represents the standard deviation of accuracy.","metadata":{}},{"cell_type":"markdown","source":"**Figure 3.**\nFrom: \"Deep Neural Networks and Tabular Data: A Survey\" by [Borisov et al, (2021)](https://arxiv.org/abs/2110.01889)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Figure 3.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.800660Z","iopub.execute_input":"2023-07-04T16:23:09.800985Z","iopub.status.idle":"2023-07-04T16:23:09.810132Z","shell.execute_reply.started":"2023-07-04T16:23:09.800957Z","shell.execute_reply":"2023-07-04T16:23:09.808860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Benchmarks for training (left) and inference (right) times for various methods on the HIGGS data set with 11 million samples. The size of the circle represents the standard deviation of accuracy.\n\nDespite the observed performance advantage of gradient-boosted tree ensembles, it is worth mentioning that deep learning-based techniques can still be competitive and even outperform traditional models for very large tabular datasets with mostly continuous properties. For instance, SAINT demonstrated superior performance over traditional machine learning algorithms in large-scale datasets like HIGGS. Nevertheless, the authors' findings align with a [study about Kaggle competitions](https://doi.org/10.1016/j.ijforecast.2020.07.007), which reported poorer performance of deep learning algorithms on tabular data compared to approaches based on decision tree ensembles (e.g., gradient-boosting decision trees).\n\n---","metadata":{}},{"cell_type":"markdown","source":"### 3.2 The Rise of Self-Supervised Learning for Tabular Data\n\nSelf-supervised learning (SSL) has emerged as a popular technique in the machine learning community for learning representations from unlabeled data. SSL allows AI models to create their own training labels by designing surrogate tasks that capture meaningful patterns or relationships within the data. In the past two years, there has been a notable surge in research on SSL for tabular data [(Ucar et al., 2021)](https://arxiv.org/abs/2110.04361). SSL offers the advantage of learning more robust and generalizable representations compared to supervised learning, which relies on labeled data.\n\n**Figure 4.**\nFrom: \"Unsupervised Representation Learning by Predicting Image Rotations\" by [Gidaris et al.(2018)](https://arxiv.org/abs/1803.07728)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Figure 4.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.811606Z","iopub.execute_input":"2023-07-04T16:23:09.812627Z","iopub.status.idle":"2023-07-04T16:23:09.823132Z","shell.execute_reply.started":"2023-07-04T16:23:09.812587Z","shell.execute_reply":"2023-07-04T16:23:09.822258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Assume we have a dataset of images and the following task: We rotate each image in the dataset by a specific amount, and the model must learn the degree of rotation.\n\n---","metadata":{}},{"cell_type":"markdown","source":"One SSL method specifically designed for tabular data is Masked Encoding for Tabular Data (MET), introduced by [Majmundar et al., 2022](https://arxiv.org/abs/2206.08564v1). MET involves masking a portion of the input data and training the model to predict the masked values. This task encourages the model to learn the relationships between different features in the data. The popular [Masked Autoencoder (MAE)](https://arxiv.org/abs/2111.06377) approach in vision-based SSL served as inspiration for MET, which focuses on masking input features and predicting the masked values. [Majmundar et al., 2022](https://arxiv.org/abs/2206.08564v1) claim that their method surpasses existing approaches on various benchmark datasets.\n\n**Table 1.**\nFrom: \"MET: Masked Encoding for Tabular Data\" by [Majmundar et al., 2022](https://arxiv.org/abs/2206.08564v1)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Table 1.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.824870Z","iopub.execute_input":"2023-07-04T16:23:09.825458Z","iopub.status.idle":"2023-07-04T16:23:09.838003Z","shell.execute_reply.started":"2023-07-04T16:23:09.825426Z","shell.execute_reply":"2023-07-04T16:23:09.837134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"MET demonstrates SOTA performance on all tested tabular datasets, surpassing current SOTA methods by up to 9%. By conducting tests on carefully prepared, simple datasets, the authors provide additional insights into the effectiveness of MET in tabular data analysis.\n\n---","metadata":{}},{"cell_type":"markdown","source":"### 3.3 The Development of Automated Machine Learning (AutoML) Tools\n\nAutomated machine learning (AutoML) tools have transformed the modeling process for tabular data by automating various stages, including data preprocessing, feature engineering, algorithm selection, hyperparameter tuning, and model evaluation. These tools, such as AutoGluon, mljar, and Auto-Keras, enable both experts and non-experts to efficiently build high-performing models, democratizing access to advanced machine learning techniques and streamlining model development.\n\n[Conrad et al. (2022)](https://doi.org/10.1038/s41598-022-23327-1) conducted a study evaluating 12 AutoML frameworks on 12 datasets from the materials science domain. The results demonstrated that AutoML frameworks can achieve competitive performance on regression tasks in material design, even with small datasets. However, the authors noted that the performance of AutoML frameworks can vary depending on the dataset and can be further improved through the use of feature engineering techniques.\n\n**Figure 5.**\nFrom: \"Benchmarking AutoML for regression tasks on small tabular data in materials design\" by [Conrad et al. (2022)](https://doi.org/10.1038/s41598-022-23327-1)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Figure 5.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.839396Z","iopub.execute_input":"2023-07-04T16:23:09.839980Z","iopub.status.idle":"2023-07-04T16:23:09.848704Z","shell.execute_reply.started":"2023-07-04T16:23:09.839948Z","shell.execute_reply":"2023-07-04T16:23:09.847581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A typical data mining workflow in materials design, beginning with a material dataset and ending with model testing. AutoML tools can be used to automate steps within the highlighted area. This is also applicable to other domains or fields.\n\n---","metadata":{}},{"cell_type":"markdown","source":"One notable AutoML tool is [AutoGluon](https://auto.gluon.ai/stable/index.html), which offers a range of powerful features for machine learning tasks. With AutoGluon, users can rapidly prototype machine learning solutions with just a few lines of code, leveraging state-of-the-art techniques and incorporating cutting-edge models automatically. This eliminates the need for expert knowledge in model selection or hyperparameter tuning.\n\n**Table 2.** \nFrom: \"[AutoGluon Tabular - Quick Start](https://auto.gluon.ai/stable/tutorials/tabular/tabular-quick-start.html)\"","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Table 2.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.850268Z","iopub.execute_input":"2023-07-04T16:23:09.851128Z","iopub.status.idle":"2023-07-04T16:23:09.863299Z","shell.execute_reply.started":"2023-07-04T16:23:09.851097Z","shell.execute_reply":"2023-07-04T16:23:09.862038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Showing the leaderboard function of AutoGluon's TabularPredictor enables us to assess how well each trained model performed using a test data.\n\n---","metadata":{}},{"cell_type":"markdown","source":"Additionally, AutoGluon provides easy deployment options, allowing users to seamlessly transition from experimentation to production using cloud predictors and pre-built containers. The tool is highly customizable, enabling users to extend its capabilities by incorporating custom feature processing, models, and metrics. This flexibility empowers users to tailor AutoGluon to their specific needs, making it applicable to a wide range of machine learning tasks.\n\n\n\nCheck this tutorial on how to use [AutoGluon for Kaggle competitions](https://auto.gluon.ai/stable/tutorials/tabular/advanced/tabular-kaggle.html).","metadata":{}},{"cell_type":"markdown","source":"### 3.4 The Development of Explainable AI (XAI)\n\nExplainable AI (XAI) has emerged as a rapidly growing field focused on enhancing the interpretability and understandability of machine learning models. Its significance lies in the ability to address critical aspects such as building trust, detecting bias, and improving model performance. By shedding light on the inner workings of complex models, XAI aims to bridge the gap between black-box algorithms and human comprehension.\n\n[Gunning et al. (2019)](https://www.science.org/doi/10.1126/scirobotics.aay7120) discuss the importance of explainability in AI applications. They aim to make the behavior of an XAI system more intelligible to humans by providing explanations. They also provide an overview of the current state of XAI research and development, including the challenges and opportunities in this field.\n\n**Figure 6.** Performance versus explainability tradeoff for ML techniques.\nFrom: \"XAI—Explainable artificial intelligence\" by [Gunning et al. (2019)](https://www.science.org/doi/10.1126/scirobotics.aay7120)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Figure 6.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.866576Z","iopub.execute_input":"2023-07-04T16:23:09.867425Z","iopub.status.idle":"2023-07-04T16:23:09.877326Z","shell.execute_reply.started":"2023-07-04T16:23:09.867393Z","shell.execute_reply":"2023-07-04T16:23:09.876383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**(A) Learning techniques and Performance vs. explainability.**\n\n**(B)** \n**Interpretable models** are ML algorithms that are designed to be easy to understand by humans. Examples of interpretable models include Bayesian rule lists, Bayesian program learning, and causal models.\n**Deep learning** is a type of ML that uses artificial neural networks to learn from data. Deep learning models can be very accurate, but they can also be difficult to interpret. However, there are a number of design decisions that can be made to make deep learning models more interpretable, such as choosing the right training data, architecture, and loss function.\n**Model-agnostic** explainability techniques are designed to work with any type of ML model, regardless of how it was trained or how complex it is. These techniques typically involve extracting features from the model's output and then using those features to explain the model's predictions.\n\n----","metadata":{}},{"cell_type":"markdown","source":"[Holzinger et al. (2022)](https://link.springer.com/chapter/10.1007/978-3-031-04083-2_2) present an overview of the key XAI methods and cover 17 ways for explaining and interpreting predictions of complicated machine learning models such as deep neural networks. They emphasize that XAI is a well-established discipline with a \"vibrant community\" that has created a number of highly successful ways for explaining and interpreting the predictions of sophisticated machine learning models.\n\n**Figure 7.** Timeline of the Developments of XAI methods.\nFrom: \"Explainable AI Methods - A Brief Overview\" by [Holzinger et al. (2022)](https://link.springer.com/chapter/10.1007/978-3-031-04083-2_2)","metadata":{}},{"cell_type":"code","source":"Image(\"/kaggle/input/tab-pic/Figure 7.jpg\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.878889Z","iopub.execute_input":"2023-07-04T16:23:09.879724Z","iopub.status.idle":"2023-07-04T16:23:09.890464Z","shell.execute_reply.started":"2023-07-04T16:23:09.879693Z","shell.execute_reply":"2023-07-04T16:23:09.889637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Figure 7 shows the timeline of the development of XAI methods. The LRP method was the first to explain the decisions of complex ML models, but the SHAP and LIME methods are among the most well-known. Initially, the methods concentrated on model analysis using either the model itself or sample data. Subsequent approaches incorporated greater and more information on the structure and relationships between the variables under consideration.\n\n---","metadata":{}},{"cell_type":"markdown","source":"**Importance of XAI** ([Based on the IBM's article on XAI](https://www.ibm.com/watson/explainable-ai))\n\n1. **Building Trust and Confidence.**\nOne of the fundamental objectives of XAI is to instill trust and confidence in machine learning models, particularly in domains where model decisions significantly impact people's lives. XAI techniques provide users with insights into how the models arrive at their predictions, allowing for transparency and accountability. By offering explanations, XAI enables users to evaluate the rationale behind the model's decisions, leading to increased trust and acceptance.\n\n2. **Detecting Bias.**\nThe detection and mitigation of bias in machine learning models are crucial for ensuring fairness and equity. XAI techniques play a vital role in identifying biases by illuminating the factors influencing the model's predictions. By highlighting discriminatory patterns and relationships within the data, XAI empowers stakeholders to address and rectify biases, fostering more equitable decision-making processes.\n\n3. **Improving Model Performance.**\nXAI goes beyond interpretability by actively contributing to model performance improvement. By uncovering areas where models may be making mistakes or producing unreliable predictions, XAI techniques offer opportunities for model refinement and optimization. The insights gained from explanations generated by XAI can guide feature engineering, data preprocessing, and hyperparameter tuning, ultimately enhancing the overall performance of machine learning models.\n\n---","metadata":{}},{"cell_type":"markdown","source":"## 4. Conclusions\n\nThe journey through the historical overview of tabular data and the exploration of key developments in AI for tabular data over the past two years has shed light on the evolving landscape of data analysis and its profound impact on decision-making. Here are the key conclusions drawn from this paper:\n\n1. **Deep learning:** While deep learning models have shown promise in tabular data analysis, particularly with the emergence of architectures designed specifically for tabular data, they still lag behind gradient-boosted tree ensembles in terms of performance and training time. Future research should focus on developing more efficient and effective deep learning architectures tailored for tabular data, enhancing their competitiveness in real-world applications.\n\n2. **Self-supervised learning (SSL):** The rise of self-supervised learning has introduced a novel approach to learning representations from unlabeled tabular data. Techniques like Masked Encoding for Tabular Data (MET) have demonstrated superior performance compared to existing methods, showcasing their potential for capturing meaningful patterns and relationships within tabular data. Further exploration and refinement of SSL methods for tabular data can unlock new possibilities for unsupervised learning and improve model performance.\n\n3. **Automated machine learning (AutoML) tools:** The development of AutoML tools has revolutionized the modeling process for tabular data, democratizing access to advanced machine learning techniques and reducing the time and effort required for model development. AutoML frameworks like Autogluon have showcased their capabilities in automating various stages, from data preprocessing to model evaluation. However, the performance of AutoML frameworks can vary depending on the dataset, emphasizing the importance of feature engineering techniques to improve their effectiveness.\n\n4. **Explainable AI (XAI):** XAI has emerged as a crucial field for making machine learning models more transparent and interpretable. XAI techniques facilitate building trust and confidence in models, detecting and mitigating biases, and improving overall model performance. They enable users to understand the decision-making process and identify the factors influencing predictions. However, there is ongoing research into developing effective evaluation metrics for XAI techniques, particularly in terms of human interpretability, model accuracy, and fairness.\n\n---","metadata":{}},{"cell_type":"markdown","source":"## 5. Recommendations\n\nBased on the key findings and conclusions drawn from this paper, the following recommendations are proposed for advancing AI in tabular data analysis:\n\n1. **Continued research on deep learning architectures:** Researchers should invest efforts in developing more efficient and effective deep learning architectures specifically designed for tabular data. With gradient-boosted tree ensembles, these architectures should try to close the performance gap, making it possible for deep learning models to do a great job of analyzing tabular data in the real world.\n\n2. **Exploration of SSL techniques:** Further exploration and experimentation with self-supervised learning techniques for tabular data can enhance our understanding of unsupervised learning in this domain. Researchers should investigate new surrogate tasks that capture meaningful patterns and relationships within tabular data, leading to more robust and generalizable representations.\n\n3. **Advancements in AutoML frameworks:** AutoML frameworks have shown great potential for automating the modeling process for tabular data. Future developments should focus on addressing the performance variations observed across different datasets by integrating advanced feature engineering techniques. This will ensure that AutoML frameworks consistently deliver competitive results, regardless of the dataset's characteristics.\n\n4. **Grow XAI:** Advance research in explainable AI (XAI) by developing robust evaluation metrics. Work towards standardizing evaluation methods that measure human interpretability, model accuracy, and fairness. This will facilitate more comprehensive comparisons between different XAI techniques and promote their wider adoption in practical applications.\n\n5. **Integration of AI-driven techniques:** The integration of AI and ML techniques, such as reinforcement learning and generative modeling, holds promise for advancing tabular data analysis. Researchers should explore the application of these techniques to improve predictive modeling, explainability, and decision-making in the context of AI for tabular data.\n\n---","metadata":{}},{"cell_type":"markdown","source":"## Links/References\n\n1.\tAnalytical Engine | Description & Facts. (n.d.). Encyclopedia Britannica. https://www.britannica.com/technology/Analytical-Engine\n2.\tAutoGluon. (n.d.). AutoGluon. https://auto.gluon.ai/index.html\n3.\tBobbitt. (2022, March 25). What is Tabular Data? (Definition & Example) - Statology. Statology. Retrieved July 1, 2023, from https://www.statology.org/tabular-data/\n4.\tBojer, C. S., & Meldgaard, J. P. (2021, April). Kaggle forecasting competitions: An overlooked learning opportunity. International Journal of Forecasting, 37(2), 587–603. https://doi.org/10.1016/j.ijforecast.2020.07.007\n5.\tBorisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2021, October 5). Deep Neural Networks and Tabular Data: A Survey. arXiv.org. https://doi.org/10.1109/TNNLS.2022.3229161\n6.\tConrad, F., Mälzer, M., Schwarzenberger, M., Wiemer, H., & Ihlenfeldt, S. (2022, November 11). Benchmarking AutoML for regression tasks on small tabular data in materials design - Scientific Reports. Nature. https://doi.org/10.1038/s41598-022-23327-1\n7.\tEgypt Pictures: Karnak Temple. (n.d.). Mrs. Redd’s Social Studies. http://www.reddsocialstudies.com/egypt-pictures-karnak-temple.html\n8.\tFree Vector | Blue 3d particles background design. (n.d.). Freepik. https://www.freepik.com/free-vector/blue-3d-particles-background-design_3086334.htm\n9.\tGunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., & Yang, G. Z. (2019, December 18). XAI—Explainable artificial intelligence. Science Robotics, 4(37). https://doi.org/10.1126/scirobotics.aay7120\n10.\tHe, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2021, November 11). Masked Autoencoders Are Scalable Vision Learners. arXiv.org. https://arxiv.org/abs/2111.06377v3\n11.\tHolzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022, April 17). Explainable AI Methods - A Brief Overview. Explainable AI Methods - a Brief Overview | SpringerLink. https://doi.org/10.1007/978-3-031-04083-2_2\n12.\tHow to use AutoGluon for Kaggle competitions. (n.d.). AutoGluon. https://auto.gluon.ai/tutorials/tabular/advanced/tabular-kaggle.html\n13.\tIBM100 - Relational Database. (2004, January 1). IBM100 - Relational Database. https://www.ibm.com/ibm/history/ibm100/us/en/icons/reldb/\n14.\tJohn Graunt - Wikipedia. (2014, July 27). John Graunt - Wikipedia. https://en.wikipedia.org/wiki/John_Graunt\n15.\tMajmundar, K., Goyal, S., Netrapalli, P., & Jain, P. (2022, June 17). MET: Masked Encoding for Tabular Data. arXiv.org. https://arxiv.org/abs/2206.08564v1\n16.\tMicrosoft Excel 1.x. (n.d.). WinWorld. https://winworldpc.com/product/microsoft-excel/1x\n17.\tUcar, T., Hajiramezanali, E., & Edwards, L. (2021, October 8). SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning. arXiv.org. https://arxiv.org/abs/2110.04361v2\n18.\tWhat is Explainable AI (XAI)? (n.d.). What Is Explainable AI (XAI)? | IBM. https://www.ibm.com/watson/explainable-ai\n\n---","metadata":{}},{"cell_type":"markdown","source":"## Acknowledgments\n\nThe author used the following tools for writing:\n\n1. ResearchRabbit: Citation Mapper Tool\n2. ChatGPT: Paraphraser and Brainstormer Tool\n3. Bard: Paraphraser and Brainstormer Tool\n4. Quillbot: Grammar Checker, Citation Generator, and Summarizer\n\n---","metadata":{}},{"cell_type":"markdown","source":"## Submission and Peer-feedback","metadata":{"_kg_hide-input":false}},{"cell_type":"code","source":"import pandas as pd\nsubmission=pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission.head()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.892234Z","iopub.execute_input":"2023-07-04T16:23:09.892826Z","iopub.status.idle":"2023-07-04T16:23:09.911301Z","shell.execute_reply.started":"2023-07-04T16:23:09.892795Z","shell.execute_reply":"2023-07-04T16:23:09.910571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.loc[0]['value']='Tabular and/or time series data'\nsubmission.loc[1]['value']='https://www.kaggle.com/code/kobbiemanrique/2023-kaggle-ai-report-tabular-data'\nsubmission.loc[2]['value']='https://www.kaggle.com/code/pawanthapaliya/tabular-and-time-series-data/comments#2326723'\nsubmission.loc[3]['value']='https://www.kaggle.com/code/poojach7611/tabular-and-time-series-data/comments#2326738'\nsubmission.loc[4]['value']='https://www.kaggle.com/code/ashishjagdishsharma/the-chronicles-of-tabular-and-time-series-analysis/comments#2326774'\nsubmission.head()","metadata":{"_kg_hide-output":false,"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.912538Z","iopub.execute_input":"2023-07-04T16:23:09.913019Z","iopub.status.idle":"2023-07-04T16:23:09.925029Z","shell.execute_reply.started":"2023-07-04T16:23:09.912990Z","shell.execute_reply":"2023-07-04T16:23:09.924013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv',index=False)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-04T16:23:09.926555Z","iopub.execute_input":"2023-07-04T16:23:09.927196Z","iopub.status.idle":"2023-07-04T16:23:09.940744Z","shell.execute_reply.started":"2023-07-04T16:23:09.927156Z","shell.execute_reply":"2023-07-04T16:23:09.939674Z"},"trusted":true},"execution_count":null,"outputs":[]}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}