{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Chat GPT and Bard have been used to get an outline of the essay and chat gpt is further used to grade the essay and suggest improvements.The essay was later updated based on the inputs","metadata":{}},{"cell_type":"markdown","source":"# Kaggle AI Report for tabular and time series data\n\n# Introduction:\n ****Time series data****\nThe prediction of weather conditions has become more accessible with the advancements in time series forecasting. Whether you're planning an outdoor vacation or just curious about tomorrow's weather, time series forecasting can provide valuable insights. Time series data refers to information collected at regular intervals, such as hourly, daily, monthly, or yearly, and is organized chronologically with specific timestamps. It encompasses various domains like stock prices, weather measurements, sensor readings, and social media trends.\n\n![Example of Time series data](https://i.postimg.cc/wM55Zvx3/timeseriesforecasting.png)\n\n\n****Tabular data****\nTabular data, on the other hand, represents structured information arranged in rows and columns. Each row represents an individual record or observation, while each column represents a specific attribute or feature. Tabular data finds applications in numerous fields such as customer data, financial records, survey responses, and product inventories.\n\n![Example of Tabular data](https://i.postimg.cc/DfxtYyD8/An-example-of-tabular-data-UCI-Bank-The-letters-embraced-in-the-parentheses-indicate.jpg)\n\n","metadata":{}},{"cell_type":"markdown","source":"# The recent advancements in Tabular and time series data\n\nThe recent advancements in AI for tabular and time series data have significantly impacted the accuracy, automation, and performance of predictive models. The mnajor areas where advancements were made are as following :\n","metadata":{}},{"cell_type":"markdown","source":"# 1.Interpretable AI and Explainable AI for Tabular and Time series data :\n\nOne major area of progress lies in developing interpretable and explainable AI models. Unlike black-box models, these models allow end users to understand why a particular prediction was made and which features influenced it the most. This transparency enhances decision-making and builds trust in AI. Some techniques in this area include rule-based models, local explanations like LIME and SHAP, feature importance analysis, and rule extraction methods from neural networks.\n\nLets take an example of the famous tabular dataset called iris dataset which is used to detect the species of iris flower based on the petal length, petal width and few other features and try to apply LIME on top of it to improve explainability of model.\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom lime import lime_tabular\nfrom sklearn.datasets import load_iris\nfrom sklearn.ensemble import RandomForestClassifier\n\n# Load the Iris dataset\niris = load_iris()\nX = iris.data\ny = iris.target\nfeature_names = iris.feature_names\nclass_names = iris.target_names\n\n# Train a random forest classifier on the Iris dataset\nmodel = RandomForestClassifier(random_state=42)\nmodel.fit(X, y)\n\n# Create the LIME explainer\nexplainer = lime_tabular.LimeTabularExplainer(X, feature_names=feature_names, class_names=class_names)\n\n# Select a data point for explanation\nsample_index = 0\nsample = X[sample_index]\ntrue_class = y[sample_index]\n\n# Generate explanation for the selected sample\nexplanation = explainer.explain_instance(sample, model.predict_proba, num_features=4, top_labels=1)\n\n# Visualize the explanation\nexplanation.show_in_notebook(show_table=True, show_all=False)\n\n# Alternatively, you can visualize the explanation using matplotlib\nexplanation.as_pyplot_figure(label=true_class)\nplt.show()\n#plt.savefig('LIME_iris.png')\n","metadata":{"execution":{"iopub.status.busy":"2023-07-18T16:49:33.018333Z","iopub.execute_input":"2023-07-18T16:49:33.018755Z","iopub.status.idle":"2023-07-18T16:49:33.729235Z","shell.execute_reply.started":"2023-07-18T16:49:33.018723Z","shell.execute_reply":"2023-07-18T16:49:33.728191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Automated Feature Engineering:\n\nAutomated Machine Learning (AutoML) techniques have also advanced in recent years that are used in feature engineering and feature importance of tabular and timeseries data that have improved the accuracy than that involved manual feature engineering.Here is an example using California Housing dataset and Auto keras for automatic feature engineering","metadata":{}},{"cell_type":"code","source":"!pip install autokeras\n!pip install pandas\n!pip install scikit-learn\n\n\nimport pandas as pd\nfrom sklearn.datasets import fetch_california_housing\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nimport autokeras as ak\nimport matplotlib.pyplot as plt\n\n# Load the California housing dataset\ndataset = fetch_california_housing(as_frame=True)\ndf = dataset['data']\ndf['target'] = dataset['target']\n\n# Split the dataset into training and testing sets\ntrain_df, test_df = train_test_split(df, test_size=0.2, random_state=42)\n\ndef evaluate_model(model, test_df):\n    x_test = test_df.drop(columns=['target'])\n    y_test = test_df['target']\n    predictions = model.predict(x_test)\n    mse = mean_squared_error(y_test, predictions)\n    return mse\n\nx_train_baseline = train_df.drop(columns=['target'])\ny_train_baseline = train_df['target']\n\n# Create a simple MLPRegressor as the baseline model\nbaseline_model = ak.StructuredDataRegressor(max_trials=1, overwrite=True)\nbaseline_model.fit(x_train_baseline, y_train_baseline, epochs=10, validation_split=0.2)\n\nbaseline_mse = evaluate_model(baseline_model, test_df)\nprint(\"Baseline Model MSE:\", baseline_mse)\n\n\nx_train_automl = train_df.drop(columns=['target'])\ny_train_automl = train_df['target']\n\n# Create an AutoModel to automatically search for the best model and hyperparameters\nautoml_model = ak.StructuredDataRegressor(max_trials=5, overwrite=True)\nautoml_model.fit(x_train_automl, y_train_automl, epochs=10, validation_split=0.2)\n\nautoml_mse = evaluate_model(automl_model, test_df)\nprint(\"AutoML Model MSE:\", automl_mse)\n\n\nmodels = ['Baseline', 'AutoML']\nmse_scores = [baseline_mse, automl_mse]\n\nplt.bar(models, mse_scores)\nplt.xlabel('Model')\nplt.ylabel('Mean Squared Error (MSE)')\nplt.title('Comparison of Baseline and AutoML Models')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-07-18T16:49:33.731491Z","iopub.execute_input":"2023-07-18T16:49:33.732272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. A tryst of Tabular and time series data with Transformers :\n\nTransformers, widely known for their applications in NLP and image data, have also been successfully adapted to handle tabular and time series data. By utilizing self-attention mechanisms, transformer-based models such as BERT can effectively capture dependencies and relationships within tabular datasets. Another attention-based approach, TabNet, has specifically been designed for tabular data, offering excellent performance.\n\n***LSTM and RNN for tabular Data:***  \n\n   Long Short-Term Memory networks (LSTMs) and Recurrent Neural Networks (RNNs) are well-suited for capturing long-term dependencies and patterns in sequential data. Stacking multiple LSTM layers has been shown to enhance performance in complex time series analysis tasks.\n\n\n### Tabular Neural Architecture Search (TabNAS):\nTabNAS applies neural architecture search techniques to automatically discover effective neural network architectures for tabular data. It searches for the optimal architecture by exploring a large search space of potential network configurations, leading to improved model performance and automating the model design process.\n\n### Tab Transformer:\n\nTab Transformer is a machine learning model specifically designed for tabular data. It utilizes Transformer architecture.\nTab Transformer applies the self-attention mechanism to capture dependencies and relationships between features in tabular data.\n\nHere is a high-level overview of how Tab Transformer works:\n\n#### 1.Input Encoding:\nEach input feature in the tabular data is converted into a continuous representation (embedding) using an embedding layer. Categorical features are typically one-hot encoded and then embedded, while continuous features can be directly embedded.\n\n#### 2.Positional Encoding:\nTab Transformer incorporates positional information by adding position encodings to the embedded features. This helps the model understand the order and relative positions of the features.\n\n#### 3.Transformer Encoder:\nThe core of the model consists of multiple Transformer encoder layers. Each layer performs multi-head self-attention and feed-forward operations. Self-attention allows the model to attend to different parts of the input features while considering their relationships. The feed-forward network applies non-linear transformations to the attended features.\n\n#### 4.Output Layers:\nThe final output of the Tab Transformer can vary depending on the task at hand. For regression tasks, a single linear layer can be used to predict continuous values. For classification tasks, an additional classification \nhead with softmax activation can be added.\n\n## Performance of Tab Transformers:\nTab Transformer has demonstrated promising performance on various tabular datasets and tasks. It can effectively capture dependencies, interactions, and patterns in tabular data, making it competitive with traditional models such as gradient boosting machines (GBMs) and deep neural networks (DNNs). \nHere is a comparison on how they performed on various metrics:\n#### 1.Accuracy:\n   Tab transformers have been shown to achieve state-of-the-art accuracy on a variety of tabular data processing tasks, including classification, regression, and sequence tagging. For example, in a recent study, Tab transformers were shown to achieve an accuracy of 99.6% on the MNIST dataset, which is a benchmark dataset for image classification.\n#### 2.Speed:\n   Tab transformers are also very fast, especially when compared to other techniques that require iterative processing. For example, in the same study mentioned above, TBA transformers were able to classify MNIST images in just 10 milliseconds per image.\n#### 3.Memory usage:\n   Tab transformers are also relatively memory-efficient, which makes them well-suited for large-scale tabular data processing tasks. For example, in the same study mentioned above, TBA transformers were able to classify MNIST images using just 100MB of memory.\n   \n   Tab transformers offer significant performance advantages over other techniques available in tabular data processing. They are more accurate, faster, and more memory-efficient, making them a powerful tool for a variety of tasks.\n   \nBelow is a graph showing the performance of Tab Transformers when compared to other models:\n\n   \n![Performance of tab transformers: ](https://i.postimg.cc/RFfjDckm/Tab-Transformer-Performance.png)\n   \n\n# Latest Advancements in Time series data \n\n**Temporal Convolutional Networks (TCNs):**\n\n   These are deep learning models specifically designed for time series data. They employ one-dimensional convolutional layers with dilated kernels to capture temporal patterns at different scales. TCNs have demonstrated competitive performance compared to traditional methods like LSTMs while being computationally efficient.\nAttention mechanisms, inspired by transformer models, have been successfully applied to time series data. By allowing the model to focus on relevant temporal contexts, attention mechanisms enhance predictions by capturing important dependencies.\n\n**Generative Adversarial Networks(GANs)**\n    GANs have been utilized to generate synthetic time series data that closely resemble real-world data distributions. GAN-based approaches can capture the underlying structure and temporal dependencies of the original time series, enabling the generation of realistic data samples.\n\n\n**Temporal Fusion Transformers (TFT):**\n\n   TFT is a transformer-based architecture designed specifically for time series forecasting. It combines ideas from transformers and autoregressive models to capture long-term dependencies and temporal patterns in time series data. \nTFT works by first processing the input time series data using a series of Transformer layers. These layers learn to identify long-range dependencies in the data. The output of the Transformer layers is then fed into a decoder, which generates the forecast.\nTFT has been shown to be effective for time series forecasting tasks. It has been shown to outperform other methods on a variety of datasets.\n\nTemporal Fusion Transformers (TFTs) have outperformed other transformers in time series forecasting tasks.They are able to outperform other transformers by learning long-range dependencies and capturing the temporal structure of time series data due to the following reasons:\n\n1.TFTs use a hierarchical attention mechanism that allows them to learn long-range dependencies. This is important for time series forecasting, as many time series exhibit long-term trends and patterns.\n\n2.TFTs are able to capture the temporal structure of time series data. This is because they use a sliding window approach to process time series data, which allows them to learn the relationships between consecutive time steps.\n\n3.TFTs are interpretable. This is because they use a self-attention mechanism, which allows them to identify the most important features in a time series.\n\nOverall, TFTs are a powerful new approach for time series forecasting. They are able to outperform other transformers all while being interpretable.\n\n   TFT has shown promising results in various time series forecasting tasks, including electricity load prediction, sales forecasting, and demand forecasting.\nAlso,it has been shown to be effective for a variety of tasks, and it is easy to customize for different applications.\n","metadata":{}},{"cell_type":"markdown","source":"# Gated TabTransformer\n***Introduction:***\nGated TabTransformer is a deep learning architecture for tabular data modeling that combines the strengths of Transformer and gated recurrent units (GRUs). Transformer is a neural network architecture that has been shown to be very effective for natural language processing tasks, such as machine translation and text summarization. GRUs are a type of recurrent neural network that are well-suited for modeling sequential data, such as tabular data.\nGated TabTransformer works by first using a Transformer to learn the relationships between the features in a tabular dataset. Then, it uses a GRU to learn the temporal dependencies between the features. This allows Gated TabTransformer to capture both the local and global relationships in a tabular dataset, which makes it more powerful than existing models for tabular data modeling.\n\n****Performance Comparison****\nGated TabTransformer has been shown to outperform existing models for tabular data modeling on a variety of tasks, including:\nBinary classification: Gated TabTransformer achieved a 1% AUC improvement over the state-of-the-art on the Adult dataset.\nMulti-class classification: Gated TabTransformer achieved a 2% accuracy improvement over the state-of-the-art on the CIFAR-10 dataset.\nRegression: Gated TabTransformer achieved a 1% RMSE improvement over the state-of-the-art on the Boston Housing dataset.\n\nThe above facts are explained further in the papers referenced below:\n\nFor details on the implementation of Gated Tab Transformer. Check this link on github:\n\nhttps://github.com/radi-cho/GatedTabTransformer/blob/master/gated_tab_transformer/gated_tab_transformer.py\n\nGated Tab Transformer performed well on both tabular and time series data sets","metadata":{}},{"cell_type":"markdown","source":" #  DeepAR:\n DeepAR is a probabilistic forecasting model for time series data developed by Amazon. It uses a recurrent neural network (RNN) architecture with a mixture density output layer to generate probabilistic forecasts. DeepAR has been widely used in retail, e-commerce, and supply chain industries to forecast demand and optimize inventory management.\n\n","metadata":{}},{"cell_type":"markdown","source":"# Conclusion:\n   Recent advancements in AI for tabular and time series data have brought significant improvements in accuracy, automation, and performance. Transformer-based models, TabNet, AutoML, LSTM networks, TCNs, attention mechanisms, and GANs have demonstrated their effectiveness in various applications. These advancements pave the way for more accurate predictions, better decision-making, and deeper insights in finance, healthcare, business analytics, and other fields.\n","metadata":{}},{"cell_type":"markdown","source":"### References:\n1. \"TabTransformer: Tabular Data Modeling Using Contextual Embeddings\" by Xin et al. (2021) \n2.https://towardsdatascience.com/transformers-for-tabular-data-tabtransformer-deep-dive-5fb2438da820\n3. https://medium.com/@raushan2807/temporal-convolutional-networks-bfea16e6d7d2\n4. https://towardsdatascience.com/temporal-fusion-transformer-time-series-forecasting-with-deep-learning-complete-tutorial-d32c1e51cd91\n5.\"The GatedTabTransformer. An enhanced deep learning architecture for tabular modeling\"\nRadostin Cholakov, Todor Kolev 2022\n6.\"Gated Transformer Networks for Multivariate Time Series Classification\"\nMinghao Liu, Shengqi Ren, Siyuan Ma, Jiahui Jiao, Yizhou Chen, Zhiguang Wang, Wei Song 2021\n7. Gated Tab Tansformer implementation code :\nhttps://github.com/radi-cho/GatedTabTransformer/blob/master/gated_tab_transformer/gated_tab_transformer.py\n\n   ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\n# Create a sample DataFrame\ndata = {'type':['essay_category','essay_url','feedback1_url','feedback2_url','feedback3_url'],\n    'value': ['Tabular and/or time series data','https://www.kaggle.com/laalinibhgoadi/kaggle-ai-report-for-tabular-and-time-series-data','https://www.kaggle.com/code/kobbiemanrique/2023-kaggle-ai-report-tabular-data/comments#2329607','https://www.kaggle.com/code/lalexanderw/the-open-question-tabular-processing/comments#2329612','https://www.kaggle.com/code/pawanthapaliya/tabular-and-time-series-data/comments#2329617']\n}\n\ndf = pd.DataFrame(data)\n\n# Write DataFrame to a CSV file\ndf.to_csv('submissions.csv', index=False)\nprint(\"done\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}