{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **STATE OF AI 2021-23: TABULAR/TIME SERIES DATA**","metadata":{}},{"cell_type":"markdown","source":"## BACKGROUND\n\nGradient boosting techniques for tabular data have been in a league of their own as far as predictive modelling is concerned. There, however seems to be a growing community of research on how deep learning techniques could finally claim this unconquered tabular data territory or augment boosting techniques at best. This study follows a systematic review of papers and winning Kaggle approaches that have produced interesting results. There are are a number of sub domains under which these findings will be aggregated. First, we consider predictive modelling state of the art, next, we evaluate self supervised, balancing strategy, contrastive learning,and missing value imputation discoveries, thereafter, we then look at advancements in model interpretability and fairness for tabular data. I also weigh in on the Tranfer learning for tabular data discoveries. Lastly, we take a tour on what Kagglers have come up with in recent tabular and time series competitions. We begin with a chronological review of papers under each sub domain. Do keep in mind that some papers discuss approaches that span across more than one sub domain. The relative performance of these paper proposals on a number of problems is proof of their potential and utility.","metadata":{}},{"cell_type":"markdown","source":"## METHODOLOGY\n\nA search for 'Tabular' on Arxiv returened more than 1500 results with majority of the results coming in 2023. This is likely due to a relative dip in tabular data modelling advancements resulting from the Generative AI boom. I narrowed down these results based on my preference for the sub-domains listed in the background section. Many of these papers introduce novel ideas founded on decades of previous research. I try to summarize their findings as best as I can. For the Kaggle discussions, I focus more on techniques that improve predictive and memory performance. These techniques and approaches can be very niche as tabular data, I beleive is inherently untenable to generalized modelling schemes. So, for good measure, I also provide a short description of the competition.","metadata":{}},{"cell_type":"markdown","source":"## 1: PREDCITIVE MODELLING SOTA","metadata":{}},{"cell_type":"markdown","source":"**MUDDLING LABEL REGULARIZATION: DEEP LEARNING FOR TABULAR DATASETS 29 JUNE, 2021**\n\n[Karim et al](https://arxiv.org/abs/2106.04462) propose a three part regularization scheme that significantly improves the performance of deep learning on Tabular Datasets. In addition to label muddling by random permutations of the label, they incorporate a differentiable closeform ridge regularisation on the last hidden layer during training. The label muddling discourages the model from learning a relationship with the corrupted labels while maximizing relationships with the true targets. This effectively prevents the model from outright target memorization. The last step of this regularization scheme is structured dithering which adapts to the spectral structure of the last hidden layer. The ridge regularization approach is more robust than the addition of a regularizing constraint or penalty to the final layer because the weights of the output layer are a direct function of the last hidden layer, thus amplifying the effect of the regilarizer.\n\n\n![Screenshot 2023-07-16 235753](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/29e5d0f5-9651-4031-a67c-ff7c6d0065e4)","metadata":{}},{"cell_type":"markdown","source":"**TABGNN: MULTIPLEX GRAPH NEURAL NETWORK FOR TABULAR PREDICTION. AUGUST 2021**\n\n[Xiawei Guo et al](https://arxiv.org/abs/2108.09127) leverage relationships between data samples by creating a multiplex graph neural network to learn meaningful representations and then concatenate this embedding with a feature level embedding. The resulting representation is then fed into a final predictive model. This picture gives a high level summary of the concept behing TabGnn\n\n![Screenshot 2023-07-17 000541](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/9e20cbf1-8bac-4d68-b811-d6e0a744e7bc)\n\n![Screenshot 2023-07-17 000652](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/6b9234cd-7f8a-4c95-8c87-5c49c7279a3f)","metadata":{}},{"cell_type":"markdown","source":"**PROBABILISTIC GRADIENT BOOSTING MACHINES FOR LARGE SCALE PROBABILISTIC REGRESSION 6th June, 2021**\n\n[Oliver et al.](http://https://arxiv.org/abs/2106.01682) propose a framework to extend the capability of boosted models to perform probabilistic inferences. Many real world applications reqiure confidence estimates on predictions. Their approach works well with the well known boosting mechanism. They accomplish this probabilistic boosting creating stochastic leaf weights. This allows the model to learn a probability distribution over leaf weights. They assume that the gradient and hessian of the boosting loss function are random variables characterised by their means, variance and covariance.PGBM achieves a 10% improvement on point forecasting perfoemance on taks with complex differentiable loss functions and a 300% improvement on probabilistic predictive tasks.\n\n![Screenshot 2023-07-17 001043](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/57e27a9f-3d35-4dee-afe1-c0b1191aa7be)\n\n![Screenshot 2023-07-17 000945](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/b0771ce0-8de9-4e68-960f-2f27a045321e)","metadata":{}},{"cell_type":"markdown","source":"**WEIGHTED AND PRUNING BASED ENSEMBLE DEEP RANDOM VECTOR FUNCTIONAL LINK NETWORK FOR TABULAR DATA CLASSIFICATION 21 JAN 2022**\n\n[Malike et al](https://arxiv.org/abs/2203.11316) in their study of Random Vector Functional links rreiterate the idea behind this special kind of network. Instead of using back propagation to train, these networks frequently use a close-form solution to optimize model parameters. This combats local minima, learning rate sensitivity and slow convergence and gives us speed, direct links and a much simpler architecture. [Quishi et al](https://arxiv.org/pdf/2201.05809) identify sme drawbacks in this model and its variants and propose novel ways to solve them. To address training set perterbutations and seeding, which result in different performances at every realization, Deep Random Vector Functional links and Ensemble Deep Random Vector Functional links were proposed in prior works. The former is a deep version of the model which allows for more hidden layers, while the latter considers each hidden layer as a classifier to arrive at an ensemble. The authors introduce a batch normalization step at each layer to combat hidden feature divergence in deep layers. They also point out the degradation of the Ensemble Deep Random Vector Functional links' performance as the sample weighting is done on the same training set because this networks pass all of the data forward in a single pass as opposed to a bagging system where classifiers are fit on different samples to improve robustness. Their new weighting scheme weights samples dynamically based on the performance of the classifier in previous layers. This biases the network for each sample to boost ensembling accuracy. Additionally, they propose a dropout system for useless neurons that propagate uninformative features through the network while training. They compare their performances with other state-of-the-art deep feedforward neural\nnetworks (FNNs) on 24 tabular UCI classification datasets. The experimental results illustrate the\nsuperior performance of their proposed methods.\n\n![Screenshot 2023-07-17 001415](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/1e4f03f2-e652-4c4b-8805-ced13ee2d5ed)\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"\n#### SAINT: IMPROVED NEURAL NETWORKS FOR TABULAR DATA VIA ROW ATTENTION AND CONTRASTIVE PRE-TRAINING. 15 Mar, 2022**\n\n[Gowthami et al.](https://arxiv.org/abs/2106.01342) propose a novel framework for leveraging between feature attention and bewteen sample attention to learn rich representaions of the data. Given an n by d set of data points were n is the number of samples and d is the number of features, embeddings of dimension e are produced for each feature d. The matrix n by d by e is then reshaped to a matrix n by d.e, afterwhich the between sample ebeddings are generated. This approach differs from [Tabbie](https://arxiv.org/pdf/2105.02584.pdf) where the between sample embeddings are trained independently of the between feature embeddings.\n\n![Screenshot 2023-07-17 001917](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/3042b8e6-d4bc-4e0d-9f94-e8bd393efa7c)\n\n![Screenshot 2023-07-17 001841](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/17612859-28c5-4733-ac2c-26e6f1459daf)\n\n\n\n\nThe authors also present a contrastive self supervised learning pipeline that leverages [CutMIx](https://paperswithcode.com/method/cutmix) to augment samples in the input space and use [mixup](https://paperswithcode.com/method/mixup) in the embedding space, producing a difficult self supervised learning task. This pipeline effectively pretains the model for downstream supervised tasks by optimising a constrastive loss and Denoising loss. The contrastive loss pushes latent representations of similar views together whilst doing the opposite to dissimilar views. Denoising aims to predict the data from a very noisy view. ","metadata":{}},{"cell_type":"markdown","source":"**XBNET: AN EXTREMELY BOOSTED NEURAL NETWORK. 21 APRIL 2022** \n\n[Tushar Shukar](https://arxiv.org/abs/2106.05239://) proposes a boosted gradient descent for Tabular data to increase its interpretability and performance. This framework fits a decision tree at each of the neural network layers. At training instance, the networks aren't randomly initialized but take on the feature importance values of the fiited boosting algorithm. In the next layer, the boosting algorithm is again fitted on the outputs of this layer. The feature importance values are again computed as the weights and passed on to the next layer. During back propagation, the weights are updated via the usual deep learning loss(cross entropy for classification) and the process continues until convergence. This method allows the network to infer feature importances and thus explainability. At inference, the boosted trees are removed. The picture below shows the proposed training and backprop scheme\n\n![Screenshot 2023-07-17 002141](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/1ad1d292-db28-4521-a5c8-cb32c4fee818)\n\n![Screenshot 2023-07-17 002210](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/c3c2d5e2-a3c8-4643-ae07-0a19628535da)\n\n","metadata":{}},{"cell_type":"markdown","source":"**SELF ATTENTION BETWEEN DATAPOINTS: GOING BEYOND INDIVIDUAL INPUT-OUTPUT PAIRS IN DEEP LEARNING 1 February, 2022** \n\nIn an architecture with many similarities to that of SAINT's, [Jannik K et al.](https://arxiv.org/pdf/2106.02584) consider the matrix n by d where d represents the feature vecture plus the target. First, the n by d matrix undergoes a positional and categorical/numerical encoding before being projected to a fixed embedding vector length e. During training, a stochastic target and feature masking is applied to the features and the targets. The model is then trained to minimize the loss on the target and feature predictions by the formula: Losstotal = (1-lambda)Losstarget+(lambda)lossfeature. The second term of the loss function acts a regularizer that penalizes the model more for feature imputation errors and places less premium on the target loss which deters the model from overfitting. The interesting about this discovery is that the model learns to use the information from the targets to learn both the target and feature imputation objectives since they are wholly treated as the values to be predicted. At test time, embeddings learned help associate the test data point with its closest target and sample neighbours. This solution however becomes computationally intensive at >8, 000 samples.\n\n![Screenshot 2023-07-17 002605](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/009e644f-cd7c-440b-a86f-56e8ee541a5a)\n\n![Screenshot 2023-07-17 002632](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/7f797e1f-0171-4861-ae98-6569a1d4a654)\n\n","metadata":{}},{"cell_type":"markdown","source":"**DANETS:DEEP ABSTRACT NETWORKS FOR TABULAR DATA CLASSIFICATION AND REGRESSION 7, SEP 2022**\n\n[Jintai et al](https://arxiv.org/abs/2112.02962) introduce an abstract layer (ABSTLAY). Abstlay learns to group correlative input features and produce higher level features as an abstraction for a semantic representation over features. By building a block containing Astlay units, a family of Deep Abstract networks can be constructed by stackin these blocks. A shortcut path is introduced at every layer within this heirarchy to allow access to the inputs regardless of meta-feature level. This subsequently enriches the interaction between inputs and learned features. Essentially, features are grouped correlationally at the first layer and higher level features are learned with access to the input at every heirarchy level. The feature correlations are produced by a learnable mask vector over the feature vector. The higher level feature abstracting function are fully connected layers with an attention mechanism and a final softmax that facilitates the final prediction for the task. These images put things into perspective\n\n![Feature aggregations for a health dataset](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBxMTEhUSExQWFhUWFyAYFhgYGBceGxsYFxgeHyAYGxgaHCggGx0mHR4dIzIhJiktLi4uGSAzRDMtNygtLisBCgoKDg0OGhAQGzgmHyUtMC0tLS01Mi0tLS0wLTU3LS0tLS0uLy4tLS0tLy0tLS0tLS0tLTUtLS0tLS0rLS8tLf/AABEIAGICAgMBIgACEQEDEQH/xAAbAAACAwEBAQAAAAAAAAAAAAAABQIDBAYBB//EAEUQAAIBAwIFAgMFAwkGBgMAAAECEQADEgQhBRMiMUFRYQYycRQjQoGRFVJTM2JygpKjsdHhc4OTocHSB0OisrPCJGPw/8QAFwEBAQEBAAAAAAAAAAAAAAAAAAECA//EACMRAQEAAgEEAgMBAQAAAAAAAAABAhESEyExYUGRA1HhIoH/2gAMAwEAAhEDEQA/APuNFFFAUUUUBRRRQFFFFAUUUUBRRRQFFFFB5WbW3CqysSSBvv3PpVfFbbNZuKhIcocSCQQ0bdt+8beaRaizrLxKyVQQUaFQMYyi4N3A3VOmDKNuNsrPPdL4brvFsZm4shsSAu4Mr4mYGaknsAwNTfiRAUhw2Xy4oWn3GJOw8nsKUN8KXmUh71uSpQsbbOxVrZtkli43Ki0SY72/M1PjfDjp9LevGL3K5l9UCsnUQWKgq+ykkzM7E7HtW94saqWp+J2UBgJGHMkhQDbxLhlIYySquQNvkMwN6wftbVOz8sucuoRafpOM8vJjgN7RBnzeUgid+h0nDyVVsbaFhkVNrdWYbgw8ZbkEj3rV9jufvp/YP/fT/J/onsJqr1thJtyWVGBVYHMaSQpJEoQAe4ZCfMnZoODsl3ms/bYAM7dJUyrM53AdmYbSIAmJFNdLZwULM+p9yZP/ADq6sXz2bnh7RRRUUUUUUBRRRQFFFFAUUUUBRRRQFFFFAUUUUBWbXakW7b3CCcFLQIkwJgTtWmk3xDduYFAgNphF25lvbQ92CR1be+0zvEENIv3VWXVSx3CoTAGw+ZgJgnvA2qWl1we3zIIEZR8xj1GM5DY9pnxIg1DiyWyh5rYqNycsdlIbcyIWQJkgePNadKBjsSfUsZPbyfpQZ/2rb9Ln/Bu/9lVajjlm2jO3MCqJJ5V3t/ZppFeY1G5cPmX7/iuxcDKGEwRIkEGD7HcfnV1eCvarAooooCiiigKKp1F9EXJ2VVHcsQB+prH+1rZ/kw90+MEYqfpcMW/1ago07uyqxdpIBMR5E+lZ9XfurctqrMQ5OQ8gDu2UQAJGx7ztXl5biJl/JWwIm46Fl8ABERsj2AGcnbuaTWeC8TuNcc6pLY5iGxNi2XFo/wAqjiCVygRDEgRJnYdLlHKY10sN/Ef9R/lW7R3C1tGPcqCfqRSbk6gEKWVifcW/7PQ4Y/mPpWy3rsFCtZvIqgAEKLnbbYWizfqBUyssaxllNaKw2+LWCQvMUMeyscW/sNDf8q3VhsUUUUBRRRQFFFFAUUUUBRRRQFFFFAUUUUBRRRQeUUu/a9qSALpglSVs3mEqSCAyoQYII29KWj4it3Tcwa7bWznzS1p1+SNwXTqAEzG42mKsm7pLdR0TdqV6O3ee2jm9uyhjFtfIn1rPe1uMg3mJA3VQjNBJA6VQncgj6ilH7cYIUtM8qAFVgh2xbFTgCRJTGT2LL3kA3h7Z5+nTfZr38f8Au1o+zXv4/wDdrXGanjN9TauXXutacpjijQzG5CvNgZBCDbbFyAepCd4rr341aAJIvAAST9n1HYf7upZpqXaz7Ne/j/3a0fZr38f+7WrtTqVtrmxhdhsCTLEAAAAkkkgQPWqf2vb9Lv8AwL//AGVFH2a9/H/u1pd8SaW6dJqQb0g2Lk9C9uWajoONpqixsu4RCUabZU5qzKw+8TcArEjaZHcQIa67zUu2VvZu1tuibe4YFdyqSoJkT7H0rXBjmcfZ7v8AG/8AQv8AnVV9LiiWvhRIElFAliABue5JAHuRXPH4iZiioXBufLzOUNioKFVVCXyyHTIMK8xiaVarWXA2N4XA2oMLirqGYrcSW5SEqeUUbqX5rECRBC46izJ3X2e9/G/u1/zo+z3v4392v+dLuFapLSlTzILZKq6fUAJIGSgldxnk0wPmiNqc6a8rqHUyp7bEf8juPpWWmf7Pe/jf3a/50fZ738b+7X/Os3F+P2dOCbnM6SoOFq48cxgq/Ip8nt39qoGuJxJvMuYLKIQCBv3ZNoBGxM9/Q1Zjtm5abFa4t1FZwysG/CBusRuD70xrlbvFkLBkuvdKgmbYQhVJAYlgkbbHHdjBgGDVWpOqLW7lq7c67QLKOUy5KVIUFkxXJWcZbSVT0irw9pz9Ol12qFtciCQImI2WQC5kjpUHI+wPftS3T8fBeGRlRisMYGOZKBbgYyH5qskAGJSe9c9w7U27kjUvc5lgffMLV5lIZWk5tbjAhroiCAriGkTXX6Xhlu3uFlv3m3aSQSZPaWGRjud+9Zrcb6hmPUb9qW8Utq9ywjgMpdiVO4OKNEjsQDB32mD3ArPd0x5ygW3CqPmUiCYcBQMhiBIMwfw9sRQPaKQ6vTXxcZ0nFikgC2QQuMtDbzAIgHz6wRisfa7gS6uS3FVlhwgBLmyYOwJUQ5kAEdt/IdXRSF9LqHsahXLEvbZbanliCynyo7yQNzG1WC3qi85gLmwx6IwklT8uR2CqRI+dz+FSQdUUjsae+QWeeYbLLIKA5FpAEbbeD6d95quxZ1S4qsBQybRbjDMFtgO5QkH+cNtqDoKK5nQ2NaOWjuxAxDsDaJ2S0DMrJ6lume8Ov9XUy6vkGCOdJj5IHRtG265+u8H8qB5WLithrlm4ixkyMFkwJI2kwYE+YpK+h1io1u3cgFHCzywVYqwUhwJCg4kbEzMmIFXKutG4MybhhuXsPvOWvSu3/lb7929NwZS11D0NbMFSr4zDAbypYH8jVqabpxb0A2JGy+4g+v60ps2tUHYyxUmVyNokKVsiNgIbpu7Dpkjv4v0NvUh25jypBxkJAgJiSFgkk8yd42Hag2/s63/O/wCJc/7qz63gtq6htuHKmJ+8uDsQe+XqKv5N7+In/DP/AH1TqdPqSjBLyKxGzcomD6wX3pys7tyTflP7QRcFoW7hGIJfpwHcAFmbInp8A9xPeoftF4nkXJyxxyszEfP/ACkY+O8+1YruqQHUK19S6WFzWcQvz9cEnEtO48AL6ipC5ZlQDalxKAFOoeqjyPpWsceU3UyymN1Jto1nGVtFuYMQB0sz2QLh/dSbgM+NwKrtcczxw099pG/SihfYm46hvquQ96VLrLXPMXLPKCDIk2oDselVI3mFckN/NjzW3U3LIW4JthkUlhKyu3c+V+prXT9sc/S19dqypYaflnKACUuMRHzFRcRQPHzk+1e3LN0lw/2hwBPQ1q2rGBshRhcB/pMB337UxGvtEqBcQlxKAMJYeq77/lUP2nYxz51vHLDLNYyicZmMo3jvXPTe5+mPT2rashGmhmHU55RZPZ3LlifplRq+M8tQzWbmTXOWiA2S7GJyUcyI7zvIAJIABNZ9NftYwWt5KgZ5KyJAOTenfufWs2rvKbtgW3SHliJtYsi94GJZiZ2xIGxJmIPS/j9sTP5022TcNxrl62zMgm2qtbKAkdklwTcgwXYL5AgE5MRqWlBybgyG5m3Cezdc/wBmaWfarGOWdrGcSckif3ZmJjxW3Q6+0LaqbiApbUuCyyoxG7CdhuNz61MsNLMt9tJPrHKEnT3D1Y4TZkj97+Uxx/OfaoNqHVnAtOQokS1uH27IS+QP9IAbeK0DX2iVAuIS4lBkssPVd9/ypUdXYOTl7RHMKliyRlPyzPePHpUxw2ty18No12ZVGtN1jqDG1C+zDM5f1chWL7PbCymme2coxtPbtkiPn+7uhSviCZ9qycauqEZbdy2lxSJ3tQuRgG4GUkJuCexPqJrcLtklQDal90Ep1D+b6/lV6ftnn6TZbyl+WbpCrK5iwyNt2WGVyfHUwGx3817b1mrBQGwHDCWbJUw9iuTyfoxG/ep6DiFhbWXNthQ7LOawGyJxmYmN4rY2vtAspuICglwWWVHqwnYbjc+tZuOq3MprwX3uOOi5PpdQN4xAtufrFt22/wCftXtvj9tnZF3ZRIGdoMxj5QjOHB/pBRt3qt9RZe6eu2xeMN1JYY/h9R37e9YeLaq1yWZHtyGCqQbJGciAS/SO4J8wDWundeWeffweDWNKDlOMhuZtQns3XP8AZmvBrXjLkXPnxibRMfv7XIx/9XtWEXLQkMbWSCbglenbuQdwPc+tRt6uzg68xQGIZQjLJELuoncepFS4WS3a45bsmvLoaKBRWUFFFFAUUUUBRRULk+PUf40E6KxXddbUw122pnGCwHVAMbnvBBj3FWXb2IZmZQqjIk9gN5JM9tqaNqOC/wAm3+2vf/Pcpd8RaN3uWCr3FBLJC/LkQGDv0mUhGUqdjmPrTfTWOWpAO2TPv6uxY+e0k0faVxV81xaMT4OZAWDO8kiPrVm0uiLR/Cq2+XDklCDICr8qYzAG5aEmTBKAxIFNbHCVT5SV8bKg2/s1sLHfcSPb/Wp7+o/T/WrupxjLpNCtvsewxAhQAPYKB/8AwqfFP5G7/s2/9pqdy7jALKMjCz5MEwN+8A/pVd4h8rRYSV3A74tInv7H9Kl3Vmop4v8AySf7Wz/89utOr04uI1skgOpUkEgiREgjcH3qGosi4ApO0q4I9UdWHntIFWW7uUwynEwY8GJg794I/WornbPDNRddbtxikhc0MRGBlVVexFySGPcXCPAjTw34d5JBW85hcT02wG3BlsVEtMmfV39dnNq7koZWUqRII7EHsQZqFjVK5IV1YjuBvHUV339VYfVTWt1nUZbPCgghGC7Rsq9h49+5/WtOl0uBJksSAPHZZjt9TV2Rgkkbe3p+dVafUK4lHVh6jfuAR59CD+Ypu2GpK1Uv4J/Ir9W/97Vt39R+n+tZtGgVcUYEKxB8wZkg7+prLRPxA3Vvl7QdjKo6qfwsC2QBBXIFQoJgdZkist74bu3tncBASAGVWYIcthicRKObZ7/IDv2HRqgXK4SBkAWJ7AKO/fap274b5WU7A7b7N2Ox7GrLYlkKbPw4is1zN83+ZpjfGMgBChonqiepvU1vTh8AAOQAIGy9h+VbN/Ufp/rUQxiSRtPj0P1q8qnGMHGLQXR3lHYWHG/f5DTSsN5VvW3t5gqylWxO8OvrOxxMj6itW8+P0/1rNahW+kW3qEuJOV3JXkkyApYRlOMEdlgbnY7Q4rJrNKtwDKQVOSspIKtBEgj2JEHYzV1pCAASWIESYk+5gAT9AKC2iiigKKKKDBr9WyAFQCOYiH/eOqz38ZD6+3eqtfxLlKjEbM7Kx8KER3Yx3OyED6itt7Tq3cT2/wDSZH5g7g+K8vaVGADIrATGQBiQQe/qCR+ZoF1/j1tSBg5YicYAIAKzMkRGQ+viYMTucbtiem4YAOygzJ2AEySfStX7Os7fdW9oA6F2AiANtoxX+yPSorwuwFKizbCkAEYLBAMgERvBA/Sgq1XF7dtgpDliuSgKSW9lHdiBuQO23qJqXjS5MjA5BsRA23cqgJOwLEHvttW7UaK08520aRByVTIBkDceoB/KoNw60ZJtWyTIMou4YyZ23k7n3oMq8aUvhi4JyCiFlsQhMdUDZ+x9DVum4xauBmEhQJyYQCJiR59O8dxV54faknlW5Pc4LJ7eY/mr/ZHpVlvTIpyVFB33AAO5k7+53oKP2tY/ir+tU6vj2ntqXe6oURJ3PcgePc1vk77Db3/0rxvoDHv6flV7L2YNFbyLXFullyYMCqwdztJEwJ2gxVb8KRVlmUYnIubdkdIG4PRAHme9aLF9gxRhuSWXENGE7S2MZVi4nqGuC0otsUd+sEOCQu4UjGQpaJMRiGHmr332XlfRRb+Dxff7Tcu3kJfO3bUhVRSCFVkxg9MEgiQ07ncm48JNplN1mCqN7iAMh/nOsZWz5kEqIkkdqbPrLxz2C4ROK3Cd/SU6vyqIFwsqk/MuU8tyB7EuSFb2gVO5v1PqKVtXcRhqnfeQyLackHxGMAe81Y3P6vvCJ7G4EUA+0JB/M1mOiIHMtPetszQQLbYE/vPaAEd/mGJPkmq+feJcGwfuzDMXMN/sytosfeVB/KCZqtc76+old05yEtbJA6xyrXVP4mbIkHzsPHakvHfhUX7QUsxK3BcB+8thsQRyzjbgKSZJp9Y0jkgTgHXIctLsgR2N1yRPsVBqH2C2VV35z5HGLouR/WtqFWPcirq1JlZ8T6JEVQhZLq2siQEvtYcoRsUGRBJB8sW/MQa0abj+pBH3V9gq7lLS3Fb6HG0xP9EEbd6ZnTrZuFrOnUDZbqIhAcbw6DCCy+Y7qTMwsbWuMWUEcsMMhHMA7dmMAA7dtjSy05WfEc9Z45qn6A6q/eGQ84COxs4RHn5p96nqLTXCebddS34ktWVH5ZIWn6tTi5ZV0GSuyswABtTB/eOaM39YmqrmlKZYXbq491YXHVpMbB5keyR3pJYXO/Mn1HPan4S0rvmXuBnt8p2N6WKjIhgVyKGT4ge3et1nhrWrarZvW7irsBdtqWgeOby4gD1WT+8KbWTdzRHsWiCMs1W4o3U7MjWyAfbOfarTbtlVLIwBbGFtum/uIyA95p3OV/U+mSxxBgcC4tZABEdbSBT6IwZ0c+wJitgN6VOdwAbMOWjBvcsFED6elVajh9k5g23IGzAozBp9MlOY+lZv2YyECxcuJkuQVhd5QEdiPwGB8qlfpU1TldfyLr/DAyl7up+Tdn+6kKB2JxAUee1Kr3wf9qOZu37NvfloGxY5IVzuAAEbwyqeoRJgnFd1vTXbuFy/ls/3dsI7hSO1xi6Ak+QSAF9yMq3tcurlBbpiei5vJjpBkH+rV7nK+dT6Z+F6AKws3IL21/Els81OwuZFZkbBgDsY8MpOs6F0FtEuADmyxwtg44N0qAsTIEmJjLes+ra++IxAcDO24FwYmDs8rEEbFZBg+DBGu3fNzkM1pg2UkEsOW3KYEmVGQ3xG2+U+Kvf5SXX6NqKp5p9B+v8ApRWWV1FFFAUUUUBVd07fmP8AGrKhc7fmP8aBBq+Ds1y46uv3haQyEgB7VtDtIkjCfzrfq9IGsPYDRla5YYgnupEn1rXqicGgkHEwVEkGO4EGT7Qao4S7tattcnM20zyEHKOqRAjefFXlb/xnjIVNwE8xGW6QiMWCgGACwbb0ncH1B/Ir9NwC4bhW5ugFsBzEEWjaJAWS3Vg3eMZ8ya7OkfGL99bhFrMg2HMYSouBkxOWJ6sc+mSNhtvvqfksZv45XvC+F8kuxZWLW1UnE5SgI7zuIj9Kq4nwVrt0ut4opiVg7yMX3nygAHoRNW6Z7zspOaKEcuGiGORVO6AiVBcjYjpB7mnlTld7XhNacpqOBXQLrJeJYlmUbiZS6AJ2gjMCfRB22jzhvADCs8IwgQQCYFx2b5TiMlbHESAK6ulOtvXFuiM2QruFX5YDkvOBDE9ICSDsNjMVeplpOniX6/hlwaO3p7YDMgVTiAJC9zDmN9jBNFngTSrZhOoMV6jgRy5CsCJPQVJI3DfUFrwd3NtTcyyOR6lhgC5KqRA3CwCY3id+9Mqc7DhK5K38N3Ft4LeUHHDZXiDbwLRl88wZ7bCrX+HCSZu7EsYxb8RvkefBvKf93+m7it29zMbTOv3bEnllkmDiAQh6p37+AIOVZzqb/IUnmC6Lyk42yxNk3zAP3f8ABG+wYbdiadTI6eKzX8Ne41ki7jyz1dJJYgid+4kAj8/NYrHw269roEBYhW3KC1Abq3X7s/2z+fUJ/wBT/jUqTPKTUW4Y3u5hvhwn5rgJ38N5W7tGXbJ1P+7HtFqcDbMObskMrAw3i4GPnyoK/wBY1LW3tSL7bXBZhRKKjQAwlhsWLGSCIgKJAneg39UV03S8gp9oIVNyQsgg/hgkkp5WPWpzqcMTTV2uZZe2DBe2VmDtksTSPVfDrHIJdCqWUqMDKhQ0AGfDMYiIECuguH7s7kdPdRJG3cCDJ9oP0rn31Wq5VorzOYbH3gNuIuYfMQUj5x2WTvspFJlZ4aywmXk30GjFprjZs3MfKDO3SBH/AC7/AE9K0uAyMs98h29ZpKNVf5FsHmi8XgkWyQFzbdvu91wHorGVnEnZlxNmFlsCwbfEqpYzJjYKxj8qlqyaLtDwHBbSm60WrmYiQTKEEMZ3Mkwf3YHvT2RI+h/6Ul1V/UFrRTIA4i4uHaCDcJOBB6ZAgjcbZTFa+DXHa2GfOSWIDriwUt0hhA3AiYHtJiStuXlMcZj2hpRRRUaFFFFAUUUUBRRRQFFFFAUUUUBRRRQLuKWeZbe2CoJIiT6EH/pS7hHDGt3C55aiLklWkvzLual9h8o2Hf5j+b9e5+v/AEFL9Tq2XUW7eS4PbckR1ZKVggz2jLaPHfarLZNM3GW7ZuIaB3updRwuIUfMR/5qlgQNmBTIQfJ/OlN3QapLWRu5uJ7O5PVgCRAyA2YkKZHiuwXtUbrQDuBt3PYe59qszsS4SuRtcHvNbJzZWuJi2V15P3LLLe/MxMncDb1FdBftk2Sitg5twDmTi0bHM7nfye9K9Bxi45tB+zW1JZAN3Y3AxgzCpgpInbMTNNuE3Ha2r3O7AsJEEKWlQw8NjE+80yytMcJCP9n6wq2V9ZJBAFxxsxZnXICRDEBSPwoB5NU6jS6hbqKWZkZpYozAEkIvUVhY2JIbuDtvtXY1C5MGO8bVepU6c/bneGcJuW2tFnyW2IjmvsDbtg7dmGatCnaG8dqz6jTXrupcSwtjApLMLcI1pjI+UnpeIEyTO0Rt03FHJsoWGZcq52wYBQWZH2kSQogSTO0KSH1vsPpTnd7XhNacnd4RqSB94pYTB51yQxQqbgOMgFiDyx09Meauv8L1DkhriFQzFZdt5ZyDjELAYL52WukvtCkzEAmfSB3rnrHFrxNoyjI58Rlhm8s4B6YthDt+LIGNhTqVOnGjhuiu23DPcyWCCubEfMmEKdhAD9vUd/C1+DahweZcRjJI+8cxJtGAzAlRKMfaRTvS6steuoWUquBSBBAbuDuZMgencbU1qTOy7OEs05U8O1cQLw3B35rypxugLMdUF7fUdzy5iam/Cr2W12VyBh7rtst20wiZ3gXP7QH0ea+6y22KLk3ZR7kxJ9h3MbwDSThHF7zm3zBEqua4FSA1lG5kEyAbrFI/6g1edOnG3g2muWli7czJx3Ls3ULYD7t2BaTHv4rP8Q8La8SyMgYWnRZMb3IBBgHYrP5gHxWng2ru3Dd5qlMbsICpEIbasATJDEEmSNpqeu1jJeRM0CsjkgjfJQCDM9u+0eO+1Z5Xe2uM1ov/AGfqC12b4RWKlMXJ2VyTs4IWVjZdiR6U9DCAMge2+2/vtXO6Xjd43bauBgS2TBdioe6AwP4xiqHJdhMkQwhpo9WXu3VyBVCoHhg0tkI8qBiAT3OVW20mMijjWga7ctOrJFsNkrHZ5e2cTA7Qp/MLsRIrXw2w1s3M7puZPKzjsI7bAf5VW2uYajlyCmBJ2+V5QKGI+XLIxMzHjEyu0nFr5wDlAxuqoMDBlxQ3CGncAl1WNycQezVN3WjjN7PuYPUfrRXtFRpoopdZ4pbbEBlyuDK2pO7BgxU9tsgrEDvCn0IHt7iSrZ52LMI+UAF5HdYndhBkfzTQMKKXaziIt8sAZcwwpk4wFLElgCAMRt6/869Xi1giRdSJImdgQYIJ/DvtvFAwqu6NvzH+NLdbxtEUMIYZICJIIFy6LeUEdgTP5VFuOWYdmuIltcSLhYYkFSxO42gK0+gBO0GAYapgqM3SIBMsYUR5J8CqtBczRXZMSygld9pn1AP6gGqdXrbAY2btxJiWV8YxIY7yIghWO/7p9Ky6jjFq0ISSBgCAuKqr3hbJkrAKkkle+1A65Y9KUcW4mbLEcsMOS9xTluWRkXDHHzmIMyTtFaX4lbHL6yRdJCECR0qzHIxC7Ke9RuNYYNcYocehmKrIyjoMierIQvnId5FBk0nFs+XNsAXM1nqkPby6SCohoUnE7iD6U75Y9KSvr9IiG6hUgW5BtoGJtsuQKlV3UjeRtWteK2SSOYJDYn2YGCDttBIB9CR6ig38selJOI8ZWzfW0UlccmYScQQ5GwBg9EAH5p27Gto4lbNxbSuSzTELIhQDJMQAZEHz47VXf1mnW4ubpzAcVJAyneVUxJ7NIHvQe8F1ov2xcKqpydSA2QBS4V+YDftP50x5Y9KVWuK2siqsAQVk9kPNuMohsYLFlaR67T5q8cUsyBzVknEdu5xjx5zSD55ifvCQp4nxIWXUMgKMG3EzKozQBEEkKdpqm3xYtp1vBELFiuAeZYOVxRgu5JHeAO57CrtLq7TLavsFR7iiNgWg+MsZgE99huPWqb2v0R+a5ZMNG4QwQMvTbpuZT6PPYzQOEQenk/41Llj0pf8AtWyBPNWNzO0DEtM7bGVYQfKN6Gqm47Y7LcB6HefwAWghbNwpCbOp37gyJoM+o41F17KWw2ABmTO7AMcFUkqoYbiZM7ACasHFZt6e6LYC3uXkCTKc3ECTEd2A3iav1Wr05lbjoY2IbE9sj2jcDBp9MG9DVF7XWE1DJcCKyItxXIWSbmYIHTIYC1+Yj0oGbKQkqoLY7AmATHYmDH1g0lscdDtZUIi82xzSWbZSUDBRt1CJlto27yYYavXW1sXbhJdLaFnAWZGGURG8qR+tUWtZp1HLc20ZVP3Qgwp7gLiDEEEiNp3oKNRxgpYS6bS5Nba6VLEQiLlE4/MRGx7b+la+La4WLL3cQxE4qTiCd9i0HEepg9vJ2qv7bpnLZEHAkyyD/wAoA5qSu4UOOoep8VanFNO3TzVOTtbjaCwYqU3EE5Sseu3fagrvcUUPbTEQ6qxYzE3GCquwPdj3Pbb12u4ZqeaocqBuwBUkhgGgOpIBg9x6+JEGoXr2mDEMyZJMyFlSwWR27kMm3eGX1FUcP4npxli1tLauEQqVwfmW7ThlxAEHmKJ+m+9A9opVf41ZUxkCcGed8QttQSWcAhVggg+fE1bf4pZT57iruBuYklgkA+etlXbsWA7mgYUUu1fFbVuwdQWHLAyykAQTEknsB59N69scStshckLBCupO6u0QhA3DGVhSJOQ23igYUUns8eslnUuqhYKNkIdTbD5D8idvRSas/bdjPll4aQokEAsQTCkjcgAk+kH0NA0orCvFLJiLi7kKN/Jxge05JHrmv7wmrR8ZtXFDq3Syq4BDB4cEiUIkbAn6A+lAzorDp+KWbjlEuIzDuAQfCn/BlP0YGt1AUUUUBRRRQVKu579/U+grJqdTjcS3iTzJghu2Kkkle+OwE+rKPNbIO+4/T2+tUPpAX5m2YXGertvtEwRJoL1Tbz+pqLrAJ3PsD39tzSu3xteZy2KqGA5bE/OS5QCPEsBBkzkPJirdTxEiy10LIUPmJgryw2W87mVgR6gzG9Bls8fVmspg4a6mQBbtIc47Eieg+fIid438N1PNXOCu7KQSe6OVPv4neDv2FKbTWEumzctpaKW1KlmkFS14ATP4QrMCewueDIphY1thLfMW5bFvMpl/+wviVO8557EHedu9Azw+v6mo3AACd9hPc+PzrDY4vZecLqNETiCdyoYDbziZjvFFziNkghrtsgrPqGU49t4ac02H76/vCQw6L4jt3OVCtNx8Yy7DEHIyQe5CxEzPcCaeW02Hft6mk9i9pIBVrTCQ6sOqTgpVg0nLoKHvspXxFMX1SpiGdRMAbHyQB58kgD1JA7mgvdYBO5/M0nsccRmtJiwa47IRl8uGe5kgwxQxtPttWjR8QF2yt1ulLglAQ2TKyzusAgxOwntMkVma9pAuRa1ihLZEEwxQEtnPcpcBmdw00G23qcrj2sSCgRpykEOxA+h6Zj0Irdh9f1NKhf09ph1qjXoeDkMt1UNBO25VfzUelXW+LWmUOLqFT2aDG5AG8xuSAPUmO9Bfrbq20LsTA9zJJMBRv3JIA9zS7h3G1usi4uvMQMst621fwdxDRkNslYeN9C8WsOhYXbbJ3J7iMQ2R32XEhsu0b1QL2kQfNbUJ92O4ChWjFd4UZIQY2lN+1Bp0WqFxnGLKbb4mSd5QMCPPmPqD370avXLbdVcMFKO5uFhiotwSDvPYz2jaqNFeXm3bVoCUVLhaW6i5uKBJkwOXE7jx4qGl1a3jZuumOdrmW2JkBWAJVtwA0ESII277UEtFxfmcoi2wFxnWC3UvKLAlgJESoHfu6irdJxFbjlAGEFsST83LuFG2naGH5gj3Aw/a7KIj8rBWe5afpg28izuWAOykrkSD2Kt23Fi6lbdwgWYZiTKlION3E/M6kEs0wBuW8k0GxeIIdQdOJJVCxOWwIxlYmZh1P0YVm03GldLdxVJV3VQMuoC5BUlT/NORA7KCZMVZpby3rrE2WV7fSXZU7kW3xlXJ8of6p9KutcJtKVKoqlTKxkI2AjZu0ACO0CO1Bpx+v6mipcpvUfof86KBYnBADabIlrAUWhIxhFdQTKkqSrkMR3hfSK0W+GKAgbqClmIIXFnuElnggkGS0QfxHvTKigT/ALCtm3btFnKWwyhThujoUwMKOkK0CIOw3NQu8AVlVWu3SUIIb7uZVlYGMMQekCQBsT6zTuigSJ8PWgAAzgCIAwABS6twEALAhlGw2idt5rx/h+2TkHuKc85BT5oYN3T8WbEz2JkYwKeUUCfUcJNy9cdmIRktqoUwcrZuHInGQQXBEHuv61p8OWwZD3B1BiBy4JW4twfg/eXx+83tDyigVXuDI6LbZnIV3b8PVzA4ZTC/LDkbQe29F/g6PbZGdyWZHL9OWVoqVb5cdioMEQd5BBimtFAn1XA0uAy9ySht5DCcGSCnyRGXX2nIenTXl7gSMpVrlwyzPPR8zEGYwjYjYxI3pzRQKdLwdbZtkXLkW5CAlICERy/l+UQIPzbCWI2r1eDqHDC5cgXDdVJTEM2WX4ZIYsTuTB7QNqa0UCMfDtvf7y4QSpYHl74Xmuiej95yNvHvvXrcBUrbVrt1uWQVyKHdXRl2wjp5YAMTBbcliad0UCk8FQ27Vss5Fr5ScJkdjOOxA2lYME+tUD4ct8sWuZdwVWRd7eyMmIWcN8V7EyT5LU9ooEd74ctsIzuDqZxumzXHdyRKGN3I+kDeKL3w5bbP7y4M1ZGjD5blpLZAlPS2h+o9DFPKKBK/AEKNbNy4UZy+JwhWYsxK9EjrYuD3UgQQBFX6nhilzcElsUGJaFY2WZ7cnEsIdpke2xpnRQKl4Qv2VtNJVXRkOJkqHBEKWB2UHESNgBUH4EhLy9yHMlejGY6tsd8u5BkT2ApxRQIT8NphgLt0CCsjlzi1tUYfJ5xDT3kbEDarrHBERlcPckM7H5OvmXDchunsHYxjB3gyKcUUCbWcBS45uFnVzEFcBGJUjupygqYD5AZv60DgKB+YHuBssiZTty7aFN17EW0JPzSNiKc0UCQfD6YC3zLuKo1td06bTrjyx0dhAIJ6thJI2qf7Bt7dT9L5r8ux5qXW/D2a4gJ9JIECIcUUCm9wZWtpbNx4RmafuyTmGBVgUKlSHI7VLT8JCWuTzLpWRBZgWAUg45Y9QMbl8maTJM00ooElj4dtKuAZ8R2BwgEIbcxhHyHGCIjxXun+H7aOjh7kJuFlcZxdZ+XIQrkAAgAAQBFOqKBJouAJafNLl0GZbdOvpRYbo/mAyIJJbeCRRp/h5ExIuXZQKEJKdItrcUADCD03GXqnwe4mndFAo0fBEtEG2zqFMheiIFtLYXdZgLbXzPff0b0UUBRRRQFFFFAUUUUCi3wZREkwsKoB2Kq+aA7SMTtsdwPyFrcNDWXssxHMzzZYn7wkkCQRG8DbsKZUUCjiPArd4vmzxcVEcAqAVtlzHyyJ5jSQZ7RFGr4YxsraVySLtty74gxbuq52VMZhYACgf403ooFFrgdtSSGfI3ecD0yLhQoxAxjqBaQQfmMRtFY+HrYUqr3FU4lVBWFe2FAdclO8IoIMr323Mu6KBJqPh605JZn3UKfk3ACxlKdRBXIEyVJMRNa73DUZg0t8qqd5yCNkuWQJMGd/ORmaYUUCu1wlVt2bYZ/uI5b9OQhSm/TiehiO3me8Gsr/AA1bgqr3EUgDFSkbWltz1ITuiqO/4fEmX1FAp13DTcvo5MWxbZGAIliz22ggqemE8Ed68XgqBQAzGFCmSvWFbJQ0L4JPyx8x703ooEGg4AFtWw7Mbi2haZlK7rgFKfIBjImYBnz4qw/D1vPmB3VsplcBtvIjGNwQC3zQidXSKd0UCrQ8HFpmK3LnUoQA4EIqlioXpnbNoknxMxULPBEVEt53CEti0s4bIIkRhByCgEkTExEmnFFArTg6C5zFLKC4coAgTIWzbmMZ3Uwd/wAIqK8DtjHFnAUEKOkwGfPYspIggAb7ACm1FBksaMI1xgzTccO049wirA27Qo9/etdFFAUUUUBRRRQIeF3mOt1SliVUW8RJgSDMDxWz4hcrptQykgi0xBBgg4HcHwaKKDda+VfoP8KsoooCuW/8TNS9vhmpuW3ZHVJVkYqwOQ7MNxRRQdTRRRQFFFFAVznx/qHThuqdGZXFokMpIYHbcEbiiigfab5F/oj/AAqnijEWbhBghCQR3Bg715RQfPNPxG9+y7L825mdbbUtm2RU3wCszMRtFfT6KKDlvjvUuliyyMyk6vTqSpIJVrqgqSPBHcVm43q7i8T4agdwjvqA6hiFYLp5AYTBg7iexoooOyooooCiiigKKKKCi93X6/8A1auS+JdXcXiejRXdUYdShiFPV5AMGvKKDta+bf8AivxG9ajlXblv/wDD1LdDsvUrafFtj3EmD4k+tFFA/wDg3VO93iAd2YJqsUDMTiOTbMLPYSSYHrXUmiigqsdl/oj/AAo1H4f6Q/60UUCD/wAPr7vw7TO7MzsplmJLHrbuTua6WiigKKKKDmvjPUOh0WDMuWstq2JIlSrypjuPauloooOb4VqHPEtchZiiW9MVUk4qW5uRUdhMCY7wKjwzUOeKay2WYotjTlVJOILNdkhewJgT6wKKKDoLHdv6X/1FWN2oooF+Z+y5SZ5Uz5nDvPrTKiigX8GuE2FJJJjuTJ7nzSzg19jq9SpZiAEgEmB13+w8dh+goooOjrjuMau4NUVDuBlpdgxA6nu5bT5gT6wKKKBwl5vtxTI4/ZwcZMTmd47TTmiigKKKKApVxu4wNiCROoUGCdxB2PqPaiiga0UUUCb4rusuh1DKxVhZYhgSCDidwRuDTmiigKTfFt1k0l1kYqwUQVJBHUOxFFFBm+OL7JpGZGZTB3UkH5D5FP7fYfSiignRRRQf/9k=)\n\n![Mechanism](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAoGBxMSERYTFBQWGBYZGRYWGhoWGRkaGRwdGhoaHBwbGxgaKywiGh0oHRkbIzYjKCwuMTExGiI3PDcvOyswMS4BCwsLDw4PGhERHDsoISgwMDAwMDA7MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAuMP/AABEIAJEBWwMBIgACEQEDEQH/xAAbAAEAAgMBAQAAAAAAAAAAAAAABAUCAwYBB//EAEoQAAIBAgQDAwUMCAUDBAMAAAECEQADBBIhMQVBURMiYRUycYGRBhQjM0JSU3KSobHRFlRzk6Ky0tM0YmOCs0PBw6Ph4vAkRIP/xAAYAQEBAQEBAAAAAAAAAAAAAAAAAQIDBP/EACERAQEAAgICAgMBAAAAAAAAAAABAhESURMhAzEiMkFh/9oADAMBAAIRAxEAPwD6X7oHIa3lmQSSAX1GmmVCCWMHLJA0NWONuZEkAEkhVB2liAJ8JMnwBqt90Tg5UHeYd4oNWidGyhGmGG5yjcEwTVli271scy/3BWM/d94oM7FnKIJLHmSB+A0ArZkHQVnSgwyDoKZB0FZ0oMMg6CmQdBWdaMRirduM7qs7ZmAmOk0ETjS/BqNRNy2DlJUwW2lYNV2Lwag24NzW4AfhbuoysY87wFWeJxWGuLle5aYSDBddxtzqp4pYwpaxkuIALoL5XUyuS5o0zCzGogyBrVmUSypHvcJdw+UvrcZSDcuMCOyumCGJB1APqq7yDoKq7RwasGV7QZZIOddJBGmvQkeupnlOz9Na+2v51LYsiRkHQUyDoKzpQYZB0FMg6Cs6UGGQdBTIOgrOlBhkHQUyDoKzpQYZB0FMg6Cs6UGGQdBTIOgrOlBhkHQUyDoKzqDxTHCyq6Fmd1tqoDGS3XKDAABYmNgaCXkHQUyDoK8tkwJiecGR7edbKDDIOgpkHQVnSgwyDoKZB0FZ0oMMg6CmQdBWdKDDIOgpkHQVnSgwyDoKZB0FZ0oMMg6CmQdBWlsbaBg3EB8WWnv+19Lb+2tBBd7vvrs1PweVLhOVe6AbgZZ/zEW4nXRzOkVaZB0FVHDcUi3sRN23lLoVAyLqba5iWHnEnmasff8Aa+lt/bWg3ZB0FMg6CsLWJR9FdW+qwP4VuoKX3QXFQozpZdYcHtQCQdNRuSImQAeW1SLNjsrgkllYZELalIE5B1UhScx70qJLaRE907uOzynL53ezKBm7uWQzoDrJ1J2IymascYYVGbk6THics+iW9lBMpSlApSlB5VdiLype7zBZQRJAmGaYnfce0VZV4RWcseU0sunN4vH3heLI1s24VAO0SZlWa4AY+SXWM26rpBmoXvvFZABcUOLRE57fn9msMe+Qx7TMNRHp3rpcTjrVtsjHvQGgKzaEkAnKDGqn2VWcUxlm49iC57O6tw5VuCBkuKCwA1WTsdPZXPxSf1rk0HiV+YU2oJbVmUwC1yD5+8dnptq2ukCXcxqmw2d7ecowIDLqYI0Enfp41YYXGWrhIUiRyIKnadmAO1Sco6Cni/05FvYegVnSldmClKUClKUClKUClKUClKUCq/HYNnu22lQLedxvPaMhRT6Arv7asKqvdJZD2MpmO0w50JU6XrZGqwdxQTMDaKW0U7qqqYMiQAN9J9lSaUoFKUoFKUoFKUoFKUoFQ+MMRYukGCLdwgjcEKamVD4tbLWLqqJJt3AAOZKkAUHNC2BoAAPRVcxxHyVE5r2rAERmY24AI+TlHWtHFsW5Yi02xtmA6rpFydWIgloGu2VZETW7EY1w4KspBVZEpCnN3iNQXMHaR5vOa9u477DdxM/FL58bjRc2++vd19IPoqTgu0ki4q7JBAABJHeESdj+PhUG1ir2aS9vvZZEqVXS0D8rlN06cx6K8HE78LpaJIJbvKAD2cged9J3fRrU3E2u8MIuWyNDnQT6TB+4kV11cPwK+124oJBYXLZgEGIS2zxE90GdZ3ruK4fL+znn9qzjODe6FCHaZBYqDMQGIBJXqukzv1lY5wLbSuYEZcunezd0LrpqSBrprVXx6O1sHuyCYDZCSSyQFz7HScwEjLGkzVnjd7f1x9wJ/EVyZMDZdEh2zGdN9ByXMdWj5x1NSqUoFKUoFKUoKPiGIFrEO7hwpt2QGVLjLIe7IlAQD3l361Atccsdvc7zeZa07O7O9zllkf8AseldVXP47hlhMShFm2O1RkJyL51syg8dLlw/7azcdpqI3vhLrMyzqwiVZTpauagMAYmNa6hdqijhtkADsrfTzF/KpYpjjpr+PaUpWkKUpQKUpQKUpQKhcVJ7PQkS9pTBgw1xQRI2kEiptRcbZLpCxOZGE7HKwaJG0xE6786l+hH95W/m/eae8rfzfvNZRe+jT94f6ajcPxty6hdbUAPct99ipm27ISAV2JUkHmIry8c3TeLVjb9i06o4jMHaZMAIpY5tZEgNH1TUf39h3gG2xl2QhleAVdlBMjQkpIG+xip9/Bs857NppABzPMgEkDVdtT7TWtOGwQRh7UjUHPruWnzd5YmepNa4Zm4jWeJ4ZlDmVBCnvK8gMqMM3zfjFGu5Ok1Nw1u1cXMokSRqHUyDBkNBFYeTNh73s6bd7wAjzdoVdP8AKOgqRbt3FEC0g56P/wDGpcczcPeVv5v3mteKw6ohZQQwgggnkfw8Kxx+LuWbVy61oFbaM5CMWYhQSQqhdTptWwpcuKBkVQ0SS5kDQnu5RrHIxSY57LYtKUpXqcylKUClKUHlQOI482yqqoZmBOrZQAIG8HWSNI61njMeEYIAz3CJCJEx85iYCrodSRMECTpVRinvm78MtpRlbILbs5y/BznLKsHNO06VrGbumcrqbS/K936JP3jf008r3Po0/eH+mqXiT3g9vswSsnMABHnJvI07ufmPXsdAxmJK/EiYJAEiSHIAJY6Ssdee2k9eE6ceeS2wt9rd29dCKTcKEg3XgZECaArA25AVL8r3Po0/eH+mufx93ETb7OTKtmhQusrEhpI0zGMw257VkMTiCSOzgagNlJO7gc+YCnNsM0QYpwnRzyX3li79En7xv6Kn4HEdogfaZBG8FSVOvpFc9w93NpTcXK8d4dDPI8/TpO8DYXXBPiR9a5/yNWc8ZPpvHK37ROOELdtGCSQRo11YHaW9SLehWSJzCNdTE1Z4ze39cfytVZx1iLtnbKJOpJ1DIZy5HEACc3dI11E1YYwy9pRvnLepVaT6JKj/AHCuTqmUpSg13rgVSx2UEn1CarrXF7jCRhMRtPnYfn//AF8Km8S+JufUf+U1r4b5o+ov4tQafKd39UxH2sN/dp5Tu/qmI+1hv7tWVKCqvcWuqpYYPEGATAbDyYGw+F3qtM3uyuvavOXvOkJcyDDBe0SSEaMwgqzAsZdgDlgV09QLvC0ZmbvqW87s3dAxgCSEI70ADNvAAnSgg4Dit7vIcPeu5GZO1U2Ar5XdNmdTmGUZu6BMwIipflO7+qYj7WG/u1NsWVRQqgBVAAAEAAbACttBW+U7v6piPtYb+7Tynd/VMR9rDf3asqxbagj8PxguorqCAyqwDRMMJGxIqVVX7mv8Pa/ZWv5BVpQKUpQKUpQart1UUsxCgbkmAPWajeWLH0qe2ovul2tD/OT7EaqhmgSTAGprph8fKb2xlnq6dF5YsfSp7agcG4naW2wa4oPa4g66aNecg+ggg+uuXsceVgJRgxzaa6QMyzIGpQgx1MCYrd5WAOo0yuw7wlsuSAB45xWvDO2ed6dh5YsfSr7a2YbHWrhhHUkawDrHWOlcngsYLoJUEAQNfEBh/Cyn1xyNTMEfhrR/z/irA0y+LU3tZn706mlV+P4ottgmUs0SQIAA1AJJ6wdulRv0g/0j9oVzmGV+o3yiX7oP8Lf/AGN3+RqmW9h6BXP8T4sbtm7bFuC9t0BLCJZSBOnjUgceIA+CP2hV8eXScouqVTfpB/pn7QqXw/iK3ZEEMIJBjYzBBG40Pj91ZuNn3FllT6UpUUqFxHFFFAQA3HOVAdpiSzdFUAk+iBqQDNqtwA7S4947SbVvwVT3m/3OPQVRDQbsBgxaUiSzE5ndvOdubH8ABoAABAAFauI4A3CrKwDKCNRIIMHwgyB9/qsKqONXWzogZgCrMcpKkkFQNRrGp0rWO9+ky1r2x8kXfnp9lvzp5Ju/PT7LfnUFnggG44J2Bu3JPoGbWsjI3e50+Nuf1V1459uW8emWGw117ly33VyFBmI0fMubQAyI21ipXkm789Pst+dVzBUYk3HVm1M3XBOUb+drAraZic9yN/jbn50459m8ekzyRd+en2W/Op+BwgtoFmTqSdpLEk6ctTVIJ+fc/e3Pzq24VeZrQLEEgusnchXZRPjAFYymU+61jcb9RD49hnN6y6h2AMaRCnMmohGYErm1lR3YkTTimeyLjhiS6kK5jMh3yrpGWJI03XXNIrZxq4y3LLBmVZKsREd4ooBzELz0MkzsrTUni+GW4qo0wzRoYIlW1B5GuboqcZcFotmuXoVUY/CHZi40Hhk++oz8TtLmm5iRl0bvNAIPeWdiVGpAnQEiYNTL/A8RmlcU7eLJYDb7aW9furE8ExR//ZP2bPWfoutY/JON7RcbjEHZBnvZL0rLXGWASqjQT5xcAajcDc15fxtuyzKz3xlgALcYsRmCrA21LGNeXXStvEOE4tbTsMTqqOVlbMSFP+ltWmxwbEXrYFy9JKqzDLZyk7TPZeHsp7ON7bDxC3r8JiNJ3c65YmDPRgda2YLELduZFuXtnJPaHdSmkbnRwZ/HWNh4Jioj3yY+rZ/teFejguKmffJnrls/2qn5HG9pfudxz3LYD6sAO9sW0GpA0B15aeAq3qs4Rw9rOhiIAEEnYAaz4KOskmrOtzevbVe1X8custqVYqS9lJESA91FMTzhjVhUXH4QXUKEkSVIKxIKsGVhMjRgDBBBjUEaVUV3vQ/TXv3hqDdxarvcvb3BpcJjIXDE+EITUg8GxPLFXD4xYH3dlpWDcExIBIxLTqdrGpjn8DXP8k1ULC42wgVEuYkAgKglhIGiAZuRGx2MHWQatveh+mvfvDVVwPh+Ku2bbviGDlEZoWxlJdQSV+C61P8AI+K/Wn9lj+zT8jje0bB8UvjEPZBLIheCwl2hLTZQdJAzt3tTtoY1vcHjhcZlAPdCknlLCcs8yPCRUThvCWttmdg5kuWMFixGWYACjuwNANvEmrVFA2AE6n01rHevas6UpWhTe6ba19c/yNXOYniQS72ZWe4HmeucZQObdyYnUZj8nXrOL4E3UGUgMpzCdjoQQY2338Kq/I2I6Wv3j/0V2+POTHVc8sbaprfE0ZmB0CqpJaNMwBg+ojaec8pYriSpcC5S28kZdNCYAmSdPR4zpVrf9z95wQy2iD/qOPvCTyrRw3BXLqF0t21Ae5b7xdTKXGRiBk2LAkHnNdPJj2zxvSFa4krPlGgiSWgHMSgVYnnmI9Iqxwfx1r64/Bqzu8AusIK2twdLjgyCCNQk7geypeA4TcFxXuFAFMgIS0mCBJIEATPsqZfJjxvtZjdtPG/8Q31Lf4vVZj1c2ytswxgZpAygnVhIOoExodYqy41/iG+pb/F6pbLYjM0gRnOWYjs8zxtBzRHhGXnmq4frEy+612XxJPeXL5uxQgSpzEE8g0ESJPhtXqtiQIiTmQljkjLkXNlA+VnzaQB417hsZcUKLqNLZzA1IAGaNPOOsco21iT7auYjOxKjLIEEcgWmDm1MZdfurSJmGLZFzxmgFgNgTuB6NvVVp7n/AI1vqD+aqHA37kxcEFySBtAVE0AkzrmNX3uf+Nb6g/mrPyfpVw/Zf0pSvK7IfF77JZcp55ARJ2zuQiT4ZmFbcLh1t21tr5qqFHoAga1G4lq9hORuZiPBEdgfU4StnEsSbaEqAXJCIDsXYwsxrlBMnoATQacTiXuObVkgFY7S4RISQCFUbM5BB10UEEgyAazF4BbV4EPcZriszG5cZtV7Ne6p7qcpCgD7qu8DhRathASYkljuzEyzGOZYkn01liMKlwAMoMajqPQdxVxurtMpuactxDh/asjZsuTwJnvo45gb2xuDv11qMnBXywb580iVXKdWLSupyxMQPXMCOt8k2fmfxN+dPJNn5n8TfnXTyzpz8d7ctjOEG5k74XKpU5VMGSp0GaAO7sc2+kGDQcIOYntJBJ0KkiCWM+d5wzQG6KBGlXGA4Yhv3wZKg2wqkmE7gJgjUyTOtWPkiz8z+Jvzp5J0eO9qHA4fs7apmLRIzHc68+p8auOC/Ej613/katvkiz8z+JvzqTbQKAogAaADSplntccNKjjgbtbJAzKZBGXNHeTUQJmCdyBA2O1WeM3t/XH8rVB4lwy5cupcDrCBoVlaZMAjMpEKQOYbfSpeMaAjHSHSY1jN3d+erCubomUpSgjcU+Iu/s3/AJTWvhXmL9RfxapTKCIOoOhBqv8A0cwf6rh/3Nv8qCypVTe4FgUVnbDYYKoLEm1bgACSTp0qC6cKAJNnDAAEkmwsADczloOkpXNunCgCTYw4AmSbCgCN5OWoPFbHDu0wxFiwFF1jc+AAGXsby97u7Z2Qa8ytB2dK5q5h+FrqcPhwJA1w6jUkADzdySAPE1mmD4YWCjD4eWIUTYUSTsJK0HRVi21Vv6NYL9Uw/wC6t/lT9GcF+qYf9zb/ACoHua/w9r9la/kFWlarFlEGVFVQNAFAA9grbQKUpQKUpQQ8fjVtKCQSSYAG5Op56DQE+qoPl8/Rfxj8q990u1v65/lNVDGBtPh18K7fH8cym655ZWVbeXz9F/GPyqNw/irW0Km3M3Lz6Pye47jl0YVztr30oCtqe/LAoZkZhExsxyjwXWa3E4gHYtK3Ik28qscmSYgkDv6wa14sU510vl8/Rfxj8q24PjAdghUqWnKZBBIExPIwDyjSudwJuwe1gHSAI6AnUf5iR6FB51Mwfx1r64/BqZfFjJaTO7XeP4Yl0hiWVgIzKRMdCCCD7NJPU1o/R5fpLn/p/wBNbMTma6wzMAFSApjctJ032HsqNjXNtZzvqVUS5Ud4gSTyA39VebyWetu/Dfto41wg2sPeuW7r50t3HXPky5lUkZoWYkcqk2+AKQCbtyYEx2cT4dyq25xW09uHN7K6GQW5ZTI3k7MunMGtq8XtfSXR0lueXNlGu8DTkZ0J1q+W908af+j6/SXf/T/pqXgOHranLJLRJbcxMDSAAJOgHM1BwjdogcNdAbUAvrHI6EjXf11JwRYXCuZiCk94zBBjQ+v7qnPl6Lhr2sqUpVZV2Ib/APLsr/pX29YawPwc0vDPiUXlaQ3P9zyiEf7RdH+6vcUIxFluq3rfrbI8ey2fZThzZr+JPNXS36hatuPvuNQWFKUoFKUoIuHb4S7380Fe7HmdwaTznf11KqDg7oN68B2cqyA5YzaoD341npPKKnUClKUFTxXEOl20FYhTmLgQAQGQZiSjGBmiBHnbgCam451Ftsy5gRly/Ozd0L6yQNetVvG7Z7W0eW4AR2OcMkGUYRoTuI3qfjj3rI63NfUjn8QKD3A2nVYdsxnTwGndzbtHzjBNS6UoFKUoK/3QKWwt9VBJNq4AACSSUaAANSaosfxJGsXEC3yxtsoHYXwZKEARkhd/VPMma62ouIwFtzLICeux5cx6BWbjtLNua4xxNLmHuKi3iSsACxf35DzNP+2+pNOKcWtvaIUXj3k2sX/kXFJ+TygmOW5kmug8kWPox7T+dVPG7OGtXsKpKJnvEMC8Zl7G8RIJ27QJ64qcSSPOO8btXLShBdYi9hmMWL+1u/adz5nJVJ9Vb8dxS1eFtEFxpuWTrZvBYDgklmUKBA5mt+LwOHtZSUUZmVBLEDmevQMY5nTnUCzisI+XLYclwpA7swWyjQtMTz261r20suBTlYSSO7EmYlQYEkwPCrWudXjdq0pCWnUnUBigGqZlJJbRTtPI7xUvDceV2K5HEJ2kkDRI+VrvmDCBOw60hVvSqGz7oxKh7ZUnIphhpcLMroC2UMFZCJ3OumhrPDe6FHcKFcBiirOXdwTrrqPqzHrFVF3SlKBSlKCm9021r65/kaubxJv9t3Pi8giY8+X355YyzHPLyzV2mMwaXFytO8ggwQeoPrPtNQf0fX6W5/B/TXXD5JjNVzyxtu3Ktir1suzpKgLAXmcomNTl1kRrOm3PO/dvm53FhRPnDQnUSdQYnYR0M8q6f9H1+lu/wf01B4Nwg3LbNcuNmFy+gyZIypddFmV3yqJ8Z22rflxThVFdxd5GLuhyqpELsxOUzude6wH1gPGrjB/G2vrj8Gqz/R9fpbv8H9NbcHwhLbh5ZmEwWI0kQYCgCY0nxPWpl8sssJhdt2IwQZswd1JABy5dYmNGB6natbcMne7cPpFv+mp9K8+o7bqj49gWXDXnS4xdLdx1zqhXMqlhmAWSMwBjnFS7fDJAJuPMAmBbiY+rWfug/wALf/Y3f5GqZb2HoFNQ5VEHDf8AVuey3/TWeGwYQlszMSIlo0G8AKAKlUpqG69pSlVFfxjQWrnzLqH1PNon0AXCfVXnC1i7ivG6pHo7CyJ9oPsqTjMOLlt7bTDqyGNDDCNDyOtVvB8SWuy0ZntgMBt2ll2S797pHgBQXNaMVi0tgFmidtCSfQBqa3mucxfE7V+7Fps+RWUkBsst2bd1yMrCCNVJ6VcZu6TK6m1r5Zs/Ob93c/Knlmz85v3dz8q5/G8QFpkUgnPzHLvIsn1uOmgPOAdPluzlzZtACSSCBAYqT4wRy8Oorr4p25+S9LjB8RtLevN3gGNuD2dzWEA3y6xU7yzZ+c37u5+Vc3juKi0VBVu8pOukQQIPTVh/2nashxW2ZAzFgCYynkWBM9AyMJ8PRTxTs8l6dD5Zs/Ob7Fz8ql2bqsoZSCDqCK5rB4lbttbi+awkf/elW/BfiR9a7/yNWcsNLjntG44nwtkmYlgIRyQdJWVYABttQdRVhiDmu20GpUm43gMrKJ6EltOuVulReLFTdtKbXaTnIIOqxlgx0kjvctNq3LYFq4Mui3C2Yf54LZh0BVSCPBYjWebosKUpQKUpQKUqDd4rbVmX4QlTBy2rjAGAYzKpGxHtoJ1c3ZuurQDYHwt04vtiQ/Z97sykaEZcgGbTKI3BqzXjVo7dqYMGLN7Q9PN3qt4tjLV27hXFu44t3nLHsLpyjsby75dO+VHpigsuACLAgELmuZAeVvO3ZwOQyZYHSKsarl4zaOwuncaWb3LQ/J60TjNoiR2pHUWbxH8tBYV7WnDX1dFdTKsAQYI0O2h1FbqBSlKBSlKBSlKBSlKCBieIZb9uyqksys5JzBVVSoMsARmObQGJg6jSdvD8L2alZmXuvtHxlxnj1Zo9VRMVgHZ7rgr3ltIok+ajlnkxoWDFY180dYGzH4K49q6ouEl1ZVDQqqTt3kXNp66CxpWCzGu/OKzoFKUoIHG2XsLodsisjIWgkLmGXMQOQmSdgBJIFSMPdDICsxykEbabHWK1cSwS3rZtuWCsCGCkDMCCMpMTGs6RtrpIO+2pAAJJgRJiT4mIE+gUG2lKUClKUCuf4yps3VvAHKWDGJ88LkYab57WgGgz27Y3augrRibC3EKMJBEEaj2Eag+I1FBBxWW/cW3Ia0EW64GzhyRbU/OQ5HJHPKo1BIOHGrLZ0cKSoVlOUFiCSpHdGsaHaqL36/DcSoxAJw1wdmMRHdtwxa2t6PiwC7jN5vfHmgQOyUgiRqDVl1dpZuaczdshiC1p2I2JtOSNtpXTYewVguDQbWCOWlhxuZ+b11rq6VryVjxxyS2rTHILUm3pl7JiUzaxGXuzoay96J9Ad83xL766+bvqdfGrrh9sC/iCAJJtyY1MWxEnnVjTyU8ccyiZRAt3ANTAtXANTJ0C9auOE2itoBgAZdoO4zOzCfGDU6lMs7ftqYSfSk48ZuWVjWcysQjahklQrSZg7xp85asMf51n9p/47lVvHrs3balDlElmIcLEoQe0HdADKJB10keNribedNCJ0ZTuJBBG3LTWNwTWGkmlaMPcZh3kKnoSD6wRy9MHwFY8Qz9lc7Pz8j5frZTH3xQb8wiayrkTdfIRkYYMYdgSSpQp2CkMoC5hrmXVidDpBWrDGnFFbIRdVFq5dObKSQy5kAg5hAckafJ11NBfVTrf7O5eDJc1uZgVt3GBHZoJlQRuD7KhYK7jBAKsQ1wE5lGZVd3Z1BLEZVDW4J3COB8kV5Zv45LKEhnOS2rDImfMUZ3bzgD3slsDSJLGRsErBYrLnlL2txmHwV3YxHyaYDFZVYFLwm5dYfBXdmckfJ6GsBcxbFAxZYuW1fIixlCklgxJkM4APQMByJPty7iReOUEW2uK0kBlCDKj6kjIMqM4gGTcHQ0GzB4sKsFL05nPxV3YuxHyehFecOxQS2qsl4ETPwV35xPJai2MfjmW23ZqAydoRlMkhR3N+7mKkgkiO0XfKQd3FhiCqWlc9o1u6S6AqA827akgHzV7Vnif+mN4oLLg6lbFoEEEIgIIgjQaEcqm1R2nxICyDlKE3JXM+c5pCkOIUGIAU6DfWnDr15cNcGVmvILjAPJklnNtcxM+aFOUsSAwBNBeUqgwNzFqS7AlC1xmDIC8LKoEAuQuZUVog6udiSA4N2vaLmNyMt12RyxKZ2ti2pLasfg7hnlJjSKC/pSlApSlApSuLxOKxCOLtsOzjykMots6s637Ys22y6JmRAAxjSdRQdpSqJHue98T2ZaVe92ZXUk+dAkH/qFliNAI5Vov4vE2WuQGuqIVSUJb/pA3DkgEA3H7oGotnUQaDpKVWJfvTalASbZa5klVzSgAGflBcwde7Vfw/F4o3UzwULMD8G6QGa8QNToVW2gzHQ5wOc0HR0rmbnFcQj3YRimdwCbN1imWQsKI7UPkYjLsWUknMKzbiuK7xNo93LIVLgGlpy3eMl/hQohV80ggsToHR0qixuMxCtYC97MGLkWrgU962oBE/Bd1nbvHTKd4qMnFsSo+KY7E5rdwlQ+Vs+ZdGVDcydmBm+DYyIoOmpVPwjEXmu3RcHcGUo2VlDd1VOUHzQHVzBk98dNbigUpSgUpSgwdQQQRIOhB2qNgeGWbE9lbW2D8lO6vqQd0eoVMpQKVzfDcXcLBWzBGN4k5WcM3bXFdC40thAFgmJnopr21jLnvTCEvdBe2jO1tC7kCzO2VoYuVOo5GgnYDE3Xv3FbMLYhrcoVzA6HvH5rK2mhIYHUQTa1yq43GuShS4ma12YuBEKi7mtI1wDVtGdyAy5StoHUGpGBx+IcnOGXu3CQbbTmUBISR3hnS4w6h0I0IoOipUfA5+zXtCC8AtAgSd4HSpFArXa80egfhSlBspSlBh8n1VnSlApSlApSlApSlApSlApSlApSlApSlAqBwzz8R+2/8VqvKUDgPxJ/a4j/muVYUpQKUpQKUpQKUpQKUpQKUpQKUpQKUpQU6/wCHxP1sR/3qywnxafVX8BSlBupSlApSlB//2Q==)","metadata":{}},{"cell_type":"markdown","source":"**HOW TO TRAIN YOUR DRAGN: A TASK ORIENTED SOLUTION TO IMBALANCED CLASSIFICATION: 18 NOV, 2022**\n\n[leon et al]() propose a performance-oriented approach which seeks to optimize classifier performance on the task at hand rather than generate samples that resemble the minority class. I am quite skeptical about synthetic generation techniques as I fear the bias that comes with the generator's sometimes oversimplifying assumptions of the underlying data. draGAN doesn't learn to generate samples that resemble the training set but learns to generate samples that improve some classification metric score on another classifier, say a logistic regression model. The architecture is quite simple and the image below describes its core concepts succinctly. The generator is trained to produce synthetic x , y pairs. The logistic regressor classifies these samples and the fun thing is that a discrete evaluation metric over the training set like the AUC score or the F1 score can be employed and the critic/discriminator will turn into a continous loss for the generator. The critic learns how to estimate the true AUC score given the generated x, y samples. \n\n![](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAoHCBIVEhUSEhYZGBIUEhISGBkZFBkZFBQSGBkaGRkZGBocIy4lHB4uHxgZJjsnKzE0NTg1GiU7QD40Py40NTEBDAwMDw8QGA8RGjEdGCExMTE/NDExNDE0NDExMTE0MTExNDQ0PzQxMTExMTExMTQxMTE0MTExMTExMTExMTExMf/AABEIAOEA4QMBIgACEQEDEQH/xAAbAAEAAwEBAQEAAAAAAAAAAAAAAwQFAQIGB//EAEcQAAIBAwEDBwcGDAYDAQAAAAECAAMREiEEEzEFFCJBUVKSM1NhcZHR0gYjMnOT4RVCYnKBoaKxsrPB4kOCg5TC8BZUozT/xAAVAQEBAAAAAAAAAAAAAAAAAAAAAf/EABYRAQEBAAAAAAAAAAAAAAAAAAARAf/aAAwDAQACEQMRAD8A/XMq3dTxt8MZVu6njb4Zjf8AkD2LbkFVvcrUJBtvCShx6QtTNjpqwHpku3cvNTqMm6uFbEMXxW4WmxLdE4r85a+uohGplW7qeNvhnMq3dQ/52v8AwzDPymYA3om5XMAuFIu+FnFugALEs2nZeaS8pk1KNPAA1ae8YGoMkFtBYAh7nTQ9RPVA0KNQMoYdYvrxHaDJZW2H6A9b/wARlmFIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgVeeJ2P9m/ujnidj/Zv7oqu+YRSoupYkqTwIFhYjtnrZXJW7WuGdTbgcWK3/VCPPPE7r/Zv7p5O1p1K9/q3/qJcnIVBsaFUAOh1JHZc3t+uWIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiBVbyy/VN/Es7sX0T9ZV/mNOMfnl+qf8AiWd2I9E/WVf5jQizERCkREBERAREQEREBERAREQEREBERAREQEREBERAREQERECg9Iq5qIuWejDS9+1SeA01HDr9fvZdnxJc/TY3a3Aege068T+qet1V84Ps/vjdVfOD7P74RaiVd1V84Ps/vjdVfOD7P74VaiVd1V84Ps/vnDSq28oPs/7oFuJDs7kqpPEqpPrIvJoCIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgQ79bA5LYmwOQsT2DtMNXQXuyixANyND2H2GfK7f8l3ao7IV3btXAp5tTQLVSiuXRU6g020Fr58RPe0fJtyrKuHT2jaKjE6MQ6lUbIqxJUMQRpe9r9ofSLtlMllDrdGCuMhdWIDAN2GxB/TJd4L2uL9lxcf8Abj2z5PZvk7XQqxWk7KUYq7Ni7c2TZ2yOJ1BTIGxvmw04nQ+TXIR2beZsru42dc7dNhToUqRyJ1+lTJ49YgfQTy/A+oz1PL8D6jAi2XyafmL+4SeQbL5NPzF/cJPAREQEREBERAREQEREBERAREQEREBERAREQERED5hNg5QJJeogPSsQxuLpoLlOGYv6j+icq7JtyJkHvZyWRG1NMspsnQ0b6Wv/AEb3PaXfXxRz2l318UDHbZduYUjvMbbMisLgNzjE5MxxIIuV4d09tp5Ox8oFhequFlDEMQSRgSwGGlyKgsOpl/Rtc9pd9fFHPaXfXxQMKpsW2s+Gd0wpkszdE+UyFgtyb4HqsLeqT7Fsm2qUNSoGAZsxl0ccQLqAo6wTYkjX2a3PaXfXxTj7ZSsemvA9cCTZPoJ+Yv7hJ5Dsn0E/MX9wk0BERAREQEREBERAREQEREBERAREQEREBERAREQEREBKg21NNW1bG+D2yyxte1uOktzNXyK/Xp/PEDSiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICZi+RX69f58vVssTjbKxtlfG/pt1TPB+aCA/OBwccdcw+diL8PTfhrA1YkdLLEZWysL2va/Xa/VJICIiAiIgIiICIiAiIgIiICJHVqBQSeA1Mi52vY/gf3QLMStztex/A/ujna9j+B/dAsxK3O17H8D+6Odr2P4H90CzErc7XsfwP7o52vY/gf3QLMStztex/A/ujna9j+B/dAsxK3O17H8D+6dgSu4UXYgAdZNhIee0vOJ41988bQL1KYOouxt6Quh/WZUq7TVG0qlvmMbs5X6NTFiFv2Ea5dRAGuWhF7ntLzieNffPPOaF8s0yta+S3txte/CZe1co1lq1AqBkVclbBz1IOINm6TNw1AU9s87DyntDsC1LBSaWWQa65KMrEgaK1+MDY57S84njX3xz2l5xPGvvmLyly1Up1TSpornEEDpFiptk+gsVFz0Rr0T2iR1OV9rAvuBr12qFVsEuT0ciCXIFhfo+uxW/wA9pecTxr75zntLzieNffMHaOVtqGgoAEitrhUYLiagQ6DpeTXTTyi29PDyttLWG5C3rBL4VGIUOoKno2viWbK5WwgfRU9oRjZXUn0MCf1SaUNoZWRHXgXpEG1jizKPWND+uXhA7EirVQouQTqBYC5JJsJFzv8AIfw/fAtRKvOvyH8P3xzr8h/D98C1Eq86/Ifw/fHOvyH8P3wLUSjS2k5dNSqlrITxPoYdRve3b65egVeUPJt6h+8S1KvKHk29X9RLUBMqryzSWoydNsXWm7LSdqaO1rKzKLA6gnsBF7TVmKvJtdajmlWVaVWrvWDUsqiE4hgjZAWOP4ym1zx0ACfZeWKD49NVdywVGZRUbFmXRb3OqNa3ZPf4X2bEtv6WKnEnepirG9gTfQ6H2GYZ+TFS6WrDGm9KoAVewanWarawcKQQ1rkEi09J8m6gWwqrZWQooWqFp4q6lqbCpnTJDWxVsQARbpGBf2v5Q0EdaZILNvCLVKdsURXZiSwAFnUC+ut7W1l1+U9nBcGtTBp2zBqKDTubDPXo6kDXtmI3yYchsq2TNTqoWKalnpUKeRANuNAtYd63VJanIlbBqaVlC71qi/NsHIetvmR3DhrcR0cTwJvwIX25b2bMUxURnK0nADpqlR92rAkgEZaaangLk2kv4W2azHf0rKwVjvUsrG4AY30Oh09BmFs/yUdNN8pXJGPzbX6G1NtK2Jc991N7/intvNsnybZTRzqKw2cbPTS1PG9OixYZdIguSRqLAWOmugaH/kvJ/wD7ezf7in8USL8CDvr4fviBpV/KU/8AP/DKfKO0stSlSxVkq55ZfiqlmZmF9VxuL9TFb8ZoVaSsLMOu41IIPaCNQZEdiQ8S56vK1OHigZO07XXSowoIjUBTDrjTJGVnuAVNibhdONvXIqW37WxAqUV3ZKB8kbG10VyuR0UXdtQdF/TNtdhQCwyA7BUcD9TSna6glWxaoE//AEVCbF8L24em14Gdt/KjU6y0tnSm2SEqFW5bo1XYgodAGRNLXOWmtpI/KG3BSd0h0sOhUsPJdIj6RHTfo436HrmyuwUxwyHqqOP+U9c0Ttf7Wp8UDB2nlLbgLLSW53ouEcgWutMjXXpAE3towt1zlXlHbirAUwvTZVK06jNYE43BsBktmy1A4HWb/NE7X+1qfFHNF7X+1qfFA5t30V+tpfxrLUrLsqgg9IkcMndrHtAYmWYENemWFgbEFWBtfUG/CecKnfXwf3SxECvhU76+D+6MKnfXwf3SxECvhU76+D+6MKnfXwf3SxECgKVRiVqWwDXFhbO1iLi5sB2ddvbfiIFXlDyber+olm8rcoeTb1f1ErctbA1amEUi4qU36XAhGvbVWHtBgac83vwnzm18jbUyladYJ0VCAFrU2wsxFgAQHCMBiB9IaXFutyPtNmVaoCsXawZhYsyEqLcAMWN+1zp2hb5T5KarWSoGUBKVRACqsd4zKQRkDj9HiNZWPJW0BEVX6S1K7E71+klRiwF7aadH8m9xwE7yfyRtCVVd6t1uxIDNYswORII1uSOsWwFuOkOzckbVkrvVKgOWKbxjqGuHLG+tujgLLaB627Zdr+YWm5yAs7K3RQ5Kcmvq/RuNRrb0zg5K2zCwrnLEDWoWtpVy1xHEtTsSLjH0a+E+T1ZSSlaxbAs2dTLJadNCx16RODDXgHuNRJn5F2i5Za5BsQLtUI+iy6jL08f08YA8mbXZbVjcVEY5VGvgqoMbhbEEipfQXyU30lZORNsCFDWuMEphTWfEqECm5wyBDdK9zlcgyZeRdptrXub2tm9mp3JwJFrX6IyAvp+iWKHJddRXBrEmouKHJiF420Oot9G4JJAvxgT/AINftX2n3RK34JqdxP8Ad7T7ogfQREQEzF8iv16/z5pzMXyK/Xr/AD4GnERAREQEREBERAREQEREBERAhr0gylSSLi1xxHqnjcN5x/YnwyzECtzd/OP7E+GObv5x/YnwyzECtuG84/sT4Y5u3nX9ifDLMQK24bzj+xPhkdVGUXNV7XA0VCbk2FgF7SJdlfbfoj6yl/MWBBTBYlRUe4ANiiroeHFPRJtw3nH9ifDC+Vb6tP4nlmBW3Decf2J8M7LEQEStzUd5/G3vjmq95/G3vgSVUyUrci4tcGxHpBlHdthucTkCDldsQL5Zg3ve/wCLe9/RrLXNR3n8be+Oar3n8be+ES0kxUC5NgBcm7H0k9Zkkrc1Hefxt75x9mFj0n4H8dvfCrUSHZiSiE8Sik+uwk0BERAREQEREBERAREQEREBEr16xUhQLs17DgNOJJ6hw9sZVe6njb4YFiJWyq91PE3wxlV7qeJvhgWZQ24sMTpuwVZ9CWGLBgePDTXjJsqvdTxN8MZVe6nib4YTUORNUFCCuID6XAtcgBgfpa8Oz9EvCVUV1FlRAOwMQPYFnrKr3U8TfDAsxK2VXup4m+GdhWfV5cpql3BSpiz4VAVZVVsSWKhgFv8AjC4Nx2zzS+UFA3ORCjNekjh2dKpo2VMbm7iwtcnskVfkhXzR6zO70d2xITILvM7gAWA1sB2DrMhPyeTIEViHDvUp6KSrb41jp+MAzsp9B7dYG9s1dXRXQ3VhcH/vCTyvs1NlUKzFyOLEAFjxvZQAJYgJ5fgfUZ6nl+B9RgRbL5NPzF/cJPINl8mn5i/uEh229hxK36YHEp6O0doGpECxSqqwupBFyLjhcGxkszFGztcqGIvxQVMb+jHT2T1u6Pdf2VYGjEzt3R7r+yrG7o91/ZVhKytop8oAVN21zvHKZbs9HKsR1cLGivEEC8u8nU9qFQmqfmzvLL0eiPm8NQL38rfXsljd0e6/sqxu6Pdf2VYK0Ymdu6Pdf2VY3dHuv7KsFaMTO3dHuv7Ksbuj3X9lWCtC87MqtTTBsVfPE46VR0racfTNWFVv8b/T/wCUsyt/jf6f/KWYGZytn0Amdizg4EBicHxFzovStqdL2vKRrV12dqbOOelXxsU6TG+JQNoBYaBuw37Ze5VrVU3e6sS74G4vipBJf1La9uvh2SvtFeszqdnOVO1nsUsDcHr1uRpppx9ECrWfb8gAt0JW+iC13J6iDooA673MmG2VloutRl5zhlTFhdiaan6K8enmB6h+mrs9TlIgl0CsBcaoQ1ldgLBrDUqpOn9ZZ5V5Sq5INls+Rx4ArmXQWc8VGBc3HZAp7HtHKDkMNKZNgWpqGZd6VLG7aNgARYEEfqs0G5QLHPEL0NAq6gICellxLAg6aXFvTGanKeAIVS+I0IQaneXvZzqLU7dXS167detyiALKGOdO/kxZLKWuMjqbsNDxXTjAg33KfYfBT98TO3HLfefxbNOwNhfk2QxbMG9xY07q/SDXqDLpnTjprYyWp8nwURAwGD1XAKXRs6gqYutxkNLHUXmtzhvNv+x8Uc4bzb/sfFAxU+ThBUipfFqTAshLEoLFiS2rEaa6DqAOs+iEr84bzb/sfFHOG82/7HxQLM8vwPqMg5w3m3/Y+Kcau1j82/D8j4oHvZfJp+Yv7hJ5DsykIoPEIoPrAEmgRpTCiygAXJ0FtSbkz3OxATk7EDkTsQOROxATk7EBERAq/wCN/p/ub75m8optZdt3fAYMoDIqkggsGJ6WRP8Alx9M1a1ENY3IYXsRxF+I9I9E87qp3/2B74GEv4Rxsb3xYAgUR+OSp46sVsOoCxOvCeRs+3hzUJbpKoKh0OIzYtgGsuWJGp04dmm/uqnfHgHvjdVO+PAPfAwaqcoMzcQtwVGVO30XFmI1K+TPbe/UNZ6XPRZqjWVSpYWQ3GSB/oi9rbwrbXQX1mvuqnfHgHvjdVO+PAPfAxtu2ja3qHmt92ETVkULc73MDKzF9KdvxeN+MjqUNvJRyzdEjoqaakoUphjbVS997a5xvb0Td3VTvjwD3xuqnfHgHvgfNW5Z/I/+fuifS7qp3x4B752BPERAREQEREDsREBERAREQEREBERAREQEREBOREBERIpERAREQERED//Z)","metadata":{}},{"cell_type":"markdown","source":"**SMALL LANGUAGE MODELS FOR TABULAR DATA: 29 NOV 2022**\n\nThis one is quite interesting. [Benjamin L.Badger](https://arxiv.org/abs/2211.02941) challenges the notion that generating embeddings to cluster contextually similar inputs might be redundant espcially for tabular datasets. He emphasizes that small networks are good approximators for tabular data as opposed to complicated, larger ones. In his opinion, embeddings can be generated by the model itself. He runs a series of experiments to support his claim. A simple one-hot encoding suffices and allows the model the opportunity to learn its own representations. He also introduces an approach to missing data and feature representations prior to modelling. He argues that there is really no good way to represent corrupt data as per missing values, ambigous features and the likes. Instead, replacing missing features by an empty vector is enough representation for the model. For longer input sequences, He highlights the utility of transformers to generate representations that suit the task. Passing simply encoded representations through such, albeit smaller model architectures could greatly improve robust modelling capabilities.\n","metadata":{}},{"cell_type":"markdown","source":"**EXCELFORMER: A NEURAL NETWORK SURPASSING GBDTS ON TABULAR DATA**\n\nGetting robust feature representations is a major concern to many deep learning based approaches for tabular data. Targets are switched to force the model to learn important representations, while features are removed randomly to force the model to learn only with the necessary features.\n\n![Screenshot 2023-07-17 002917](https://github.com/Israel-Orere/NFT-PRICE-PREDICTION/assets/83463364/a9132571-c1bc-426f-a081-09a09039eff5)\n","metadata":{}},{"cell_type":"markdown","source":"**GATE: GATED ADDITIVE TREE ENSEMBLE FOR TABULAR CLASSIFICATION AND REGRESSION . 27 JAN 2023**\n\nGATE uses the well-known GRU units to learn feature representations from the data. A sparse learnable mask is used to select features at each stage of the network. A typical GATE block consists of an update and reset gate. At any stage, the feature representation is a linear interpolation between its previous and candidate feature representaion. The update gate determines how much information is used to update the feature reresntation at this stage while the reset gate controls how much information is to be dropped from the previous previous forget from previous feature representations. [Manu J and Harsh R](https://arxiv.org/abs/2207.08548) employ novel Differentiable Non-linear Decision Trees(DNDT) which use all of the learned features to make the splits non-linearly. Leaf responses are learnable paramneters in this architecture to add end-to-end non linearity in conjuction with the soft binning for the gating decision propopsed by [Yang et al](https://arxiv.org/pdf/1806.06988). The leaf responses make up a decision stump for each feature. The output of a single decision stump is gotten by an attention based combination of learned leaf responses. \n\n\nThe figure above illustrates this better\n![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAATAAAACmCAMAAABqbSMrAAABm1BMVEX///+6yNPh1ef62dWw4+b6+voAAADIyMjV1dWqqqqurq7S0tLNzc3Y2NixsbGKnay1vcTV2dyjr7mnt8Sstr/s7OzAwMD09PS4uLi+zdjl5eUAigClpaXf39+iACWdnZ0etv/Fsc6QkJDn2+7bzeLAq8qEhITKudLZzt7RxNjv6/Gfn59sZm+VlZXm3+ne1ePCuMhmZmh7e3uUjJiIgYx8dX/yyMOupbPr09D/5OBHQ0lYWFijmqfNwtLz5vqRztKm1tm2qp//+OZUVFQiIiJdWF9wanPy5eMwLTHktrCdiIbTt7SrlJHCqKXm8faDj5pSXmsAeQCfABCApaiVwMPJ3+E6Ojre08GUhnnh4NTMwbWWq7JTIgCYhGpvd4Spoptyf4EMJzhUm8UjmdR+psAApvF4kKiWq79ofIyCmKFIUGFxnruqwdZqc3VlbYMAkNQwPk99e29lX1eSjXyFc3FOQ0KjqZ/Z6PbE1cRMlEwohihhnmGqxKphYFlcZHp5qHmfOEGzcXWiPEVqiIpUbG43R0heeXtuY1Ounoe6OVkMAAAcGUlEQVR4nO1djUPbRpYfO5Vl2bJsbPRZeR1kIpsYEWygUDAJJiEkJNi0+QCcOyvXXgO7h9MmZUN3b3u3u2lv+2ffjD6sGeNPbPPR8BLAkp9k6ec3b96895sRANdyLddyLddyLb8TEegziXDR131hEjnbYfRwr+IKSQRQY8SOIBAV+IdS1TLapMruGxzYt15vot+fMmDrB2BTLUvx/XgoJ6rynrrvk2JgPQFEdazKVF+xQTbOMgD8W/nffaFYkKn65E8csDjYkzju2Cyndn2cxgDNrGhgvRZ7KjHSnvAiXlXLG+YJeKqasdDraqr6H+ATB6wWqqq0pnNUaDcUr5ipMJ1KgXUf2FUFw2DjKZ0rm6kyCO3th1K6YHJj8U8bsDPJNWAdhKdO77sGDBfUS/LWKz4YpoB+DRgmrQDL8yCo2y9zlJbU+NMq14Dhokm8xgEQYgFgOAZExNMq14BhEhbzoTANqDxsmirbehB0DZgnYQMkQQRGqkIE8AzPKa0OuwasT2GHexVXSEKR4Bkkol70dV+YUGeTi77sa7mW36EoksgAFTCUxNORIKsIYghuqmgzGQ7bOrQSDicFnoNv5EBIFJRwMEnzEoU2c59aw+zYQVI96Hxq0jE04LrrCC1GAAOL8s23/2l9WZLOpkZw/kEk2OlNsbvOKPAC39y8efPAepWXRnH+QaRj5YfpriONALHkdxCwb62X9/mWiaULlF6u5ryvmEeAfYNe0axx2QZgXKc3pe46I/FhB9/d/B710EJILPDGCD5gVNIDGCPxYSAS9dJv0uVqkkynN+nuOuHRIHZFx/UtsqynVEbz/V9SwPhcKB9iMMkxIVxy1m+uwOE7SY8W+aQAC0eaekCWtCknxCd1QsRW8lMDjAwMmwBz8qykDgmY8qkBRiBES2yY2Lb/EDpGysAx+rScPgSMuDI+ECAMxkGK0JECOr45GrwuMWBJYkchR2w6YyJS5wGBKf2pWRieupmZnMhkZ06rYTozU5mJzNSdkV/ZuQE2FutNbG2iSU5NTPqhTE1MNXaFmq7+zsQUUvFPTmTdXSMZGp0jYOO+niRqNysImFtwvH3L35Bbt8mTujoOXBZkt0Y7YDlXwGTZAkW2XsoORHIbwBw3NTOR9gBLu4iR6R0MUoSqfYpRpHfAOQNWqCUgUrFaQpaNes3CSpZT8Ec9gvhZEHqA2fcrTnhopRFixEltnYyf0PFPjPI+zhEw2XgjV3yvE+8OXybkvI/7H/ha1g/iPv2AOTIqcqVCAGYPdDC8inNZ+Cdj7cbTO7c9wFbni4uwVVqu/zL7MAqNUZpdR4QmaA/QwuTC/YO8byN1UJchYHI+/8dl/aBS1w6jeuVN6o9/4g5lDDBLZjLIatKWcc3NIkwmLBxwMDwVv3/67su0Y2KXOQ5jAwz/gBymiN98d/O7/8J2QAvTDuSdvJy3AZNjb1cSlcJRrh5/tay/PuAqb4UaDphV7ofOfGrj4fR0Me1Pb9jOzOoqnUjf0pnwpxc2NqanV/3+2YUFhGoGeTp2JIixoWR3pe6SY3JNe1C1wK0XWN828mH1eoKrR0MJiJwGfVisHvXVK6nxekU9KtTkSix6ysJgi0zPBvzpwEI6vXG36HfbJGa8MxDUu4G5u3NQbSE7b5khii1Gk96JBIZyGirQ9HVa1YKbN793N+/zMaeXtH6cXtLy9O6PD/3zRe0DghG7I0QuDAJ299HDdHF6bq7h0p1yEdK5A4O0dGDOPx/wrz5cLc67VjiK9M7m0tL20tLm4CcKP4CGymFFgbAN2Hemk6kKBCpk+NA2DlOt9JehWD6RsgFbWH6U9rtuygIMS++gniAd2JgPzHo6CLChp3eWvnj8xQ0k8O+AmCkCXYBXiKVXxO8twL51NiUVxDBUHGtqCVjjFJZPdCxsCqIxVbR9mAVY0tNxLOzuMvRxdr9gAzbk9M7m4xuYfPF4EA/JvAThAE0ABg4sA8NsDov05cLr+75DqynacjpwBU5CAgG2GJjNBorFrOP0LR/GejowqkgXAxur04Hi4uKiHe2juGK46R3HuDx5vHT2k/EiEEUeKEQCL/Ht99/gXTAOWL6SCP1QOKqrOyv1w0LtrdwKMAts6NCzc8U5/+JCMT2N9ZI8pgNBnV1dzaZnF9MOYBPoJOIQLYx6fOOUfPHFoGeVQp0uER9LJsZ/fnWoH9VzB++OHr6W3x21AsxqdagLtKOwtAOYHYfh6R1vHJkuLnpGOMz0Tgu8EGTD+4AWglvYz5V84k/a4Q+56Luj6XeVnZYWZvt1L9KftUIsJ9In2DveUNK2MCsMG2ZVvDVeN74YoFV2F9zCZBjFxnzOnnqCcPqN+7RT0rexsZE1+rbfD+E6M9hgEnkwOws0vMH3Umu8oB8b0WjCkrbpnTjZSzYAc2JTMlsxQeZ3HJ0sjthkZtiX3sbARtwoe8uHyR5gLjNnplU+rIm9c8ezw0YGcWgW1tbAIGDtPiMy+JCzv4wrwAfPLTKuzTrU1ERmEjbGiczwW0l7A3NMrGTU0G/rOqyOmwPBC5imj9exZ6ampoicfgv2zu072Tu4yrDSO5vNERjhxaDCOgy2TZB6b1ZemZrMHBeqb4EqXTK+3Tmyd7wW+eQGfG1vLsENt03uHwAhtR4fT4Ef4+taKFGqngCNP/+5FYOyd4aV3nEB+/LpjRtbW1vb1sb7L5/YiKFBpamflIxkak8vm5UIWxaqMhA6Xv0IJGkwHM5FCRkqwTvJ2VsSwUUhU0y9pXdoy2lSbrooLDTSXe4+q0U+2Tre3oKgPXLge7/95ZZtat6pLnR1kUaK2pXWZBSSTUFu9ZTeKcn7vv3Y7hs2KUGwKKCLYpJX6DEg7ddYUQJszAZsa/vPXx5D3P5st8XtJ8enAbtQOc3eIaf5DYu9owNTlvbohKrvpxK/nYB/pZRilUvMmSIb/6CJhlY6sQB7en/rxtaN7adP3i9tQai2t7cQfg5g1toiJfLEuyc9fPoQ5TR7hwQMS+94cgb2Tsm3ry3s0Qe51HouvncCAQT6b5xcNCNc8YO2n3tVPbEa4fH28fZT2+cfI2f25Msn2w3AGEmXQaqqyWYhmS9o42ZBKdw/ZzpnM3sHhMlNLL3jCUdYXLCnSw7SQFHEpMgDnitQ6IS8IlI8HHQleRAEvEjZXmv7yQ2nY3Q6ym0vcjUUvQxyx6qg5aUTYz1Vy/0IzpswDAEjSN1cPk9s4+kdV/L3XxIq/X7Hwdblja5xGIiZepmSSxqvGwJ0fmUtuqnH+/zwQeUUeydAgNGSvaMEiH5qaOmdrpH+aGQ82pMcNqpG5LQYhpxF4FgPqbNAtVAZXDqNJYdQD2kn426pyC0YuZlpt5REDL4hYBjjMJuZgJLBuExOzOHpzGTgYDKD6wyv8t0tW7G/k0zSZZ5VxuBXaJ4I4nFZKHc5Z3dBle9UxUIF/Y5XorKX3vFSGS0AyzRSEd7o2wHD1cEyGo1sxfBG4W1NzDGwajRvpiqmWSuA/cr9fSP6TjAqA38qxIR5q6HSR+qw7pPfV/Jv7GpkDiKpHtnFShwwtw1mJr30TsZFzBl5ODpEzsw1siGyd9qZmOPBShGfBrTU7okBTMowzXjINKOdz9iDQMCknYpPr9X0w+WEnJflvFapMbWd10btUDvYySd+qL+RT7F3CLwa2VRXbJ3bOF4QsRZUxcFEbI3YY+dtdr+8rytjQfj9bWpsVR3LVXNjHU9IqwKfa8pn/PSXv/73PrY9jhrf+E7+JJ+zuRXy9Eqllj+S6+P1Q+31gfDxLU1wK2w3dQcrcaQ9NIj0zq0GXtlZlPe3ikZDZe+0ROxx9+Payv1wsimd8bevPv/8868wxCBg3J/qh3rtberg54T8vrZyoNV2mJ2deqq2o0MLG/8f4e0pbgVW4EjPL6Ait03+wsHITjYq3v7l7HLaqcQNNdQ+jdhgVTZ2pSmdQSG8Pv/8r85mOOwwEO26rYy9bPxt5lZY30B20qM7WXj5pywTcyIul71ThLIIdVanF+ddutOQ2TtLJGSPBwwoEKeFxwYqP9mAfSU5EshHe+RWWOuwsjnHwjKwmW0EpjeWpxDdyUYM7cdGliLUWQ0srG48QhSy+VXk6RCow2bvUEuN6vcXg5S9LUnpEhAN3UNs3wHMWdokGFCJ4pDcFr1o2F4NhmqwdxDd6e7LjbR/Om2xd6yikBNPIJ0Zh1uxGMhml1cX5lxQR8De2VxChJSlpYFtNycqcOTCG9j3/lcLsP91tpJk1Uit1+S41QZPcysaZ8DYO7OzgWK6OOdQENGbp9k7CwHYJiGgDSsc0eSsIaQJKa0A6EII8AJ2iTw0sa/+gmnhle/7MvPqh1e+6Pjrin6gvo62rHyjjtABrPjwQdrv+vVm9o4N2ByikKUXZz3+xYgmZw0xr7qcJ3JWP/3tJ3wTt7BYfid2+O6onnvz7ihfOPq5JbfCZe+gJpkNLN5dnF+00Ghm76Am6Q/Mp5cfZeeLD1ft8AL0mt7pW84tEY1b2IZsRN/qRysWGaUdYFYUj5z+w8C8P/Dw4Vx6uUGWw9M70ArTc3Pzs6sLC9n0tNeTjmhG7kUA5huvV2QdtsfxRM4XLdRaN8kGM2c1i+i+U3eLlgtzOOWYTsYj+BRXpxqtdkSTsy4EMCfqsnCSNV9rQp2VuqEaA+/04pxFQbRnNuDpnZnGYGB1Y8OLPEaUIz4/wNpFEkQnKTezd6YaQ8lFy8JsAyPTO+RkEXe2yIgmNpwfYD0mEBsHOD0INjZquPwmHeoWPvhuZvgMWS7vsw3c9A4+82rSnUYUInREXOeWk6wY0eSsywtY436nGuyvU/Sdhk7G1clMjJLkBi4zYF666PbUxEQmMzEx5flxrllHnMpM3LqVmfKa41X3YX0LWdG4fbtVhaNTTzisXpLL8QqGPQ5YyKCUERt0H9KR/yJ01xlSeoelDYCvVoABFuI1itOG8ikjl/Nbe4fnCoBpbWHJUC4sXZ4m2nH9sGB3nSGNJVWBZTkMFexliuHVCyDOtRSRCZGrE4U4QnT7D7E3RNZ6B8xWRFhbwmwYCutK2MBeh9G7zsYFryXWbe2dluwd0qP1kQ+zNclPoPtb1lnBl9kQeTub1/MFDC5emc0RtiV7h9Qh6U59NMl5hNZvOaKW2CdvnMcAWz+kKyy08C1+ffDyZI/SzN4RQgK5Fo/9h9DRNeLxIH04/Tmw+7pmskMDbEfz5Uw6egx/+jvL2QVb6MMSMUBOHXZcL6EjBAhSVh9hxRyoJkIl/gDfNwhgid1EzgDF3JKQ6O8sZxeSjAJFJx2UYz2kzstWKr0Ip/LSmEJ6xAEAo8YAHWG5aFIsnxtbn6Q7UXemMuRCRM6FYDp30GJFmawH04BDowEAs2RfHZDAw8Z7EqfR44Q6KmNPnZk8PXWmoZO1VVrpnPV6+1Pnh7I6Ay79Tc7CnP7MBJa6ccfWZHoHTGCMlVuOZQ2W3glz/SHAMcMOIiyqgK+xWlFTObI5Re2Rgu/gU/vSEyQzx9EhFitKD2V5p37Xp9DyQ/hQQhAZZaf+BuLE7FRk+X195cDmVKDFinItFiuy3dTt5sWK7PeJ9A6xWFF6KCnqFy+evejHB5VfPO9LvwdBhLqdeMKoH9qLFcnyhlo/lPK11/CPlsjvxJbzB75T7J02ixXhRoRNyF1dXlxI+yezzSr9ibh27zMoa/de9KZP2fqf3Vvr9TNZiea55mH7k+0n+CYEbDymvcX4YfmVSr5+JNfj+UorfpgVRziT5C3jsleJsSd04+kdVLuEYvEEFqzi5WBlthf27VsQ9GI1/eojuc8mDRLdza+h/B3bBwFTd7TDQmXH4of9XCm8KURrqcN83ahs6NGdwqtW/LAMWqxoY3p63k8uVoQNlKxCbmB5/tF8Or04Z1VyrbzrWdM7L9Y++6wfBDC8ekfsFD8M/P0PUL7+h7Ol2KTgWEKWkfOPwf9x6zX8H0M/PvhWjHT6DvfLZu88mE778cWKnE4f6biLFS0GshDThUWE2ABUgTJ+/xCBbmdp1u/RrlGfomAd8ebXf7AQc8fyATbWqk9s0UvaxwedkNQmo2T9D+bTxfm5xcZiRdhHWZy7wHRxeSG9uri6uuha4RnTO+T9Q082ZH1bDA0GRrruobvkAObw6bjAA2Kxog78MM0Snbc6Qoe9M7fxqIm947RYpGOzd+ZXA8V0ozx5drrTC+/Wn61hJtbENtssO6OMsteAnz3r1cQoDfAPYOdSwPbZgP3T2RJoYrGi8VpFTnhzHFpYGHBSN85iRdnAbHrRWfbEAizs6aDFilCTXF5Ory5seICdjVBn3f/z58/h3482DnZXaZbBfo4qxTdzQqpcje9WZFNEVGlXH/7++BHT74hXKgXYlEQlVWws/OTrr//w9dcYEZTkh2mvfoj74rFoNJeIV/CVPpqWYZhAPV9gzh9YWFicXfBq34qngxYrWni0MVsMTM/O2svNWHyCszl9O6AIrD14vvbsOdbGIGDvzXHumD7R1l9xe9KJyZih1w5gH39ZC3z27PnHPtokwk3T8Etc+sc//4ETZ3HAmJVa7C3ihx2+P9h5d7DSEjALE8QzmV2d9WcXV+/aZBTLoRPpnQknXkUt0uaHWbzzs1WNbJcUePbr2nMCsFKcNUNC5RjN7zNqe1ylogrMuKP/7Jdnv8IW/Kw/wLoIDlhNNiuH+lHe5oe1WazIMldvBU3YA75Mu2AQU2c8Jv/i/PRUwwgH8fkB+O+z54QXp6hTZ6Rc/Wcfn//62dpaP16/u+CAqfmanI/Xa6ED46hSqFda8sNsp4pF8UWPykSmdxpDSZtm7bfZKGcbGll3/ezl81/+79ka7sM66v/yy9qvH5/11U12F2KxogY/DP4utOGH2ZGD6DFzMCoTqdO0WNFUFpxdsE7v+SnAKHfdllJjZozXq37sCeDepW16h+gk5SbKZvNEIjd3Q07OIjIaU05G7GzpHS8OdZrYPexNqgh7uFgwx/7cCOjFhr6D770WJz2TxBM9yRuSFAzQ4t2NZNfkrSYQXB1s+t+tAVeIxwdGzQYDASu9Mlgp+qE3/YGEH+tNGgd46fDshGVBGW/pdzeBSOhMTk5mMIbPGROIVNNQh3jvIRMx4iaHAwY66J+v4H3S7ezUVPb26TdxHXHmzkwLlb6FGBz2cP8ioX+BBfBB2TtnTu94CNxb60WfcgMK6PbO+JGjlx7YOwOssvli7d499N/L1RhMJ1pT+ZT+RQithIMKeuAtAw1JUAUlHEnSvECF4CbnWI/1YFyeTUbQI3S5sGY/UTcZCaPNgT6dIvOhekRXjLiUitSMfWP8qTGmxcM5n+lVjqlLtl6Y0NPTcI0hehCyzKYHK9VoLmzQvmodfEiBuJnak8ZLmmdTl4cnhoTusUo4RDog+YkMqI7l4lWGrambqZhExXLqviqbKaxuPLyPHlx6X1FweIi1/orWfe30LxNgyd7dETW0RYL6XSbzEgHWFytSHNbDpK8uYH1CoAyJdHpVARP7dktJaSiryFxRwBSt+WEj3SVIF7ordZUrChgXKPQQ1JMiPRjGg06uImAiz9/PB7vrNQnFBGh+4GkNVwuwpMSEGE6gw5GgwKGX4Z7unxcQvV9iI2FWkEIMExpgGvPVAYyS1NCpMr8Cd3Zpm6zKnMIHIsj0NKhqcbo+9S8KsEiOa2cVdK79RfEqo7R5K6h2fPRGuxP2+QwWnu3b1w5DIqmOLotvA5nYAUooVKj/laGlQH8ARAL9O9uBhUp1bQfEcjWucN2HTjm6q0qT9Bv/DmuE0Yck0Wd2fdTBqYXqRU1Ej6LqcljkAm5oxEJb98x2DeyblnbjLSSCXVNh1BWZCdqzsHajUbqnG4jBuDPSVJjuLufSIMZKkkteQ5Vozi4Pu984/Cv1kOvmncJZD/4IJLH256Ag9NITXpZW+TOvRHhlDPas4BiAY6Ua5QEPf4WBAvhSAuwe8ugh1HxEDLc/CTIsCob2othDkO5NIrBewWPQYV3jgEiHzz9P+VcuVqIPfxtTq4ktsP4GgMIW2NpNvA9W4N+qD+xGj8cKwcoefb/adklrFtmmGMiFWZY2XrbTgrjaLc9tlJRlly/zQXiYGmgPtCFYneslMbECACU6sRvTzFdbANTgfwSYrAEdASaDzYQGKkA3wXz7+ZY2Aqw9arbIsqi3xEMIipXCwF113Y2rOJv7FLAVaecw3EQpGh2m20+uCHeeDiNay8MDGlXf98uAcqZoUnDbKciLAuEqXddgDyas4YnYwwgB3kQ17NtnNXMc3ndVL5R3NX298i9DrmraZhRs+lRQ0mQT5NaJ6YN4E3IQKHgTUCB2om1qrMChACzP64Io2dcWSRKHCQH8MGcaYNg6LLXCF1iFdlxdZwcpQWgkUJWT4th6RRWBDhbgnjDPKzzcRYXLYK+cVJM8HxljRbYMSieKglwRn+RZSkmWymH0jIwgFaFyUCnM9xnK6l01XjbmHPNuXBlwo6zwigg0K6+FlrlHN5qH3wwr2F+q6Ki5J6i7GTA+n3Sm/SQj9mEFygC0YjfJztX0dd++WYat1kzkdqXxHLqDvRQXp/bHNb6orucS8L13sspFfMVUVYItqJRMmEYZmGO5ajgWZ6pxWgYfqjQtjH9ggnQ810tn15PQ9mNdufsBF1TFdce8ayscy2mijiiGAniJ8KS0B5RX8XZMpYG40yiBpBgGVbAPW3YOA95hnQGLlcwTBlSTOVMIx80yqIdjpiDF9+M0Z4RoAXoWbj9hSmqsaFS58RgwlZwE1VjGKNHje76SFFeAthutVuIfQqYUL7Whw1DObirsmiCFrq/hfyi6HdBeblB0vQHruAAJ3i+l5BV4h0HFwieUpAuc4hzhWqQLAO/yxUJghadWeGgeyaSFJ6MIeuOwzoAhVySWe1yfZrep3rZu9Fr6LuwmRAui3Qqdt17wsL3z0PsqaDkCFGe842si9KjoURtim/jSuZPGQ1CdTsAKz2kJ+XYFOXU3JUHzuJoXk9rxBm25A9Y7LOV+p5ehnp+vHu4pBRRuJUBpTDfHCqYkTZtKcU+q7I0vwB604FuvmWWtWnk/Xvkt3npeoX3nlJ3Ap70UVhO+7qxPFyjnYfV236l4cDQd5uI8ogWf+hNtV95drDE+ZKR7Uo35kADVWLUQLOpgC4YV6Av+7UW0VDaqJ1tAM8FvLR/4w1tt8qU1CeTlg26fKTagsSKrQgEdttI9k38p4rAfS/Luia7JYLeuRYGulTWdFnQjuFgy4k/BU/TksgLIs/p09eQpMEqVNo8yUCnk+yzpzlPyxpssMkZe7O0w7kJyfIPJj23vynJDqLN0Y632wmEBrUUHsQ7guoWMyctQ2OlX2luBCM0mQieDkZDYJWEhEENCDYbnHB+MhOkuFcxhlcQvjYgaSEoRNizwnV1NqGkIDYeXmsKyEaEzIOFQx7evpKTsQgLdyRmJ2ilHxDkDxY4eSh38obWXUNhct36fa2UnSqrbGhNsaojcxEslnNrBTihObQMom+rUUdBdAb3KIqhthmC0ynUwk4jaJknd9o3fj1CSyghENY2nQ0y3wjdElGEkYt6yGOTOXPi+aiKyECKGCaEfJkT3aiNURELHhKwfRvg9t8RruZbfh/w/oBKmVR8XMywAAAAASUVORK5CYII=)\n\n","metadata":{}},{"cell_type":"markdown","source":"**T2G-FORMER: ORGANIZING TABULAR FEATURES INTO RELATION GRAPHS PROMOTES HETEROGENOUS FEATURE INTERACTION 5March 2023**\n\nAs opposed to popular GNN-for-tabular data applications which consider latent, between-sample relationships, [Jiahaun Y et al](https://arxiv.org/abs/2211.16887) propose a feature-level graph estimator to learn latent, heterogenous feature interactions. They allow a static graph topology to depect underlying feature relation, data adaptive weights are learned via the graph estimator module and then higher level feature relationships can be learned. The T2G former can then aggregate learned representations across each level.\n\n","metadata":{}},{"cell_type":"markdown","source":"## 2: TOWARDS BETTTER INTERPRETABILITY","metadata":{}},{"cell_type":"markdown","source":"**LOCALGLMNET: INTERPRETABLE DEEP LEARNING FOR TABULAR DATA. 23 JULY 2021**\n\n[Ronald R and Mario V](https://arxiv.org/abs/2107.11059) 's conceptualization for deep learning interpretability relies on the simple but powerful structure of generalized linear models. Neural networks obscure the interaction between an input feature and its target. Their key idea is to maintain the GLM structure as best as possible but then allow the regression parameters in the GLM(Generalized linear Model) become feature dependent.They call this new beta function over the feature space, a regression attention. In a pass through a normal feed forward layer, regression attention parameters beta conditioned on the feature space are learned. Therafter, feature vector x is passed through the network. Finally, the LocalGLMNet scalar multiplies the parameters and x. The result is a generalized linear model whose parameters are conditioned on the distribution of the values the feature vector can take. The set up can be trained via stochastic gradient descent\n\n![GLM](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAbQAAABzCAMAAADHcAfMAAAAilBMVEX///8AAADh4eF6enqYmJj39/fl5eWBgYHu7u6NjY35+fn8/PyQkJDz8/Pe3t6tra1wcHDX19dcXFy+vr7p6enGxsajo6MjIyNlZWUzMzNHR0fOzs7R0dG2trZRUVFycnJgYGCGhoYnJycXFxc6OjpEREQdHR0NDQ2fn59VVVVFRUUlJSUuLi4SEhJBOAnjAAAJEklEQVR4nO2daZOisBZAuSCI7CgqS4MbLryx///fewkiJAjq2F1v3q2654M1aAhJTpabOGUrCkEQBDGCTuDBujkzgMCDRtLwMW2lHacagYCNKM353y+kxAfoorTZvy0L8SYmScMHSUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUMISUPIj6VZum6/m9R03036Brrp/l5miu261i9m9+4zP2uPH0rz5gBwXbyVdlVeglL960eM5Zas4jT6rdzCcuas89/K7T0W5WS3zj6582fSFgC5ogEYA59lvTZwTmxklPMPyshZLuXr2Ym9pNsPczN8uYt/AWu8C3zYB6zceyNV6MvXObAOvEk+mS5+JM3c81t8gKH+sgHpMgI+HtPiw0loXkiXITCJdvWfzzJjnU2XrpMNe1nBO20/gHn2XydifUK6tODCXqdr85MHjkjTVZU1b+R5+vB9TTn4GNsBDD1ZW0uXTl1mCJ4VxlBZv7PUodxieYhqf3jZz9qz3EyVjxzDGFg3lrK0DL74I07PGtBSVZaRO9Qe5vodaStZml83WpI+7cSmyucwQ+1VYUSawdaqwGYdGp6VJ4U96/TfdZ95QJamH7muC4S9VNa2+zeznyh2ALuB3GRp1pUPDQf6K6Q+6wZLtgf2NB8OA83SkzY7mHzwTvrJ8lX7T3PLf6TPPA1NKh9JC5K6HI/hgNO9lZV8EmNrUG8OHZEWLyfAZ4vy3i75zGmYteuBcYKpO4HTcJFlaSazH+3OPWeZa4PiNQ29DFgfCafH41B+srSQVdbYlL3cWEZpaDeLn3XSj6B5oB0HcutJuwaK5fe7p75QnJlyb8Gd78M1umpD694n0txv1gUm0FupFXepVLnSRANWweK8jQpB8d5IYy0AvOfBprkckrYEWC7hNBI7Tv+IVwv4Ustjf26Zgp9cuh6up3A8Ds8X8Va8ysELr0U/txxW8QS0toYOrE9fg7llcq+HibV5mAOiJF1Nq1MryYPzcaMMYe3FmMvz+bioAyfLFyI0X3qmAbk5/34I4L4OgTa5dnPmCsp1X+wTaTb/ndzlQ1VEJsBGztDsuOl+tvXQvLUDVpDZw+JnHqESupEGA8HAtsutuqfbs4y0pJ+bHcNeGAkZPM54fLJpMe/pFs2LTC4vDgCPUVR07XJr5tKC3xQBV3SBuH5rIjyzudHnLesP1NaBszADZ4O/nDoePRawVbZP44YCrjZ/PdelCIJWXuSpqhGcIpXRvDW/1g/jFbO7dl0WmySIu5LP4Ngo9Ir2XYPlFs0Llpt375nrrcLjdN46XthmZ8yDIii6KrPZth5o4W4ax/dbXV421mSsaF7zrFW9ZgCbgC0vbJdJ3TluttWk87S/tZQ5CbTtfWjZKsvuaz0xWG5NF/B5ya0Nj5HU9JbO5M90gLfHvVYat6ff277F1Iq5VnRdxYChYTMuzYF00c3gl6DFuQ8NqDtSWveeFUShXARpTYv+8Eq4vOeHVbc87XwLFKdd7r0DNIvNpV9YaU1z65KGbIivEn/VLgy+Y2/VRTc9zvkukpczZlNGIq4LSxDHzOYP/+xcWFHiLMp7JaKt6V+iqh3NbGdTbzv4ziCV4hH3z1+vaXaRKjzs7w+KcGNtlmHVXsftABYZl+bDXpj4ln7Loql+WGeoniHla/FecY8gNowkLaybL2dT0HQXxMIHulAVu7w002Oglc+k3YIun7VcxpIH3UfuSVgAZnF1q5Gv8slelzIQrvQTbzqLrQZmziekLlTThJKqMIGEq/aYoUCKID4IRIy6ZB5rQGO2YAGDsPSWwpS+KopbVPHlZKxP3ofQuLSwW0KG8fmq51b1mDB4L5yDuOxL0urhYJd1bDLZDlcqPU7ZFDRZ1i0VP5N2qY9gzk3xqqHdWgQ7YLPtWr8ddfly3SRpUb1s7Jo4eSdvLxuKDSz5bKvVQ3dxkHYlH0jL6k4cg21OMlgZOQzcwLoRWBdIzJO+KK7flQan+wPHT0TgMbiRmML6yIODTGmkBdKkoR2EixiOfp7GdQOOSNNgY7GawTy63SBL24odaH5O/bzc3GaufHBr/wVMgso2fnV4Ee3lwF+SlgObFTdlXfZ8/hCT1pz5pMKm21rWND1IR3TmyKZHRpLmQHJZbOt1e8F6oDm0k8jg+0sxDrD3FdsqWNtW9xyejbQXB6glbHW2wFq3mwve0OJIk9xcYzcMjYEPOiIeF9j1QYzyKG0nrpcwNdvcFvvBruXWcUF0iw7sKpZVZKWwpk2BxR/NSYe3qPqxQZ1BfTLhqs2jrEpe0+J3TszzVLgoSp09sy6EU3E/Ax1Pv1WhPtdxWfu6yQtpOrt8ekZU7wm6+kVwZHtBGDuT8cTldESaTF+aiCFE4tn1je8NpjGv0Qh2JbYmG53DW7sWy+Ix/Dtjaxx93TXunAUOs+TFDTzoUttN87C0DKaQDtwrkoEwFJkwxfzejqVdiBuSy2gygfjJ6a3fxQpRyfrh6sUh9BJmk0v/KKjF6DrUcsKlPV8U6mB9OnD89Dd8dYGMeVgo5vlxAy3j1N+m3JtkWJoPDzuIB2bSOXGUxHE82nhalzRj901fVTlj2/NkNdbMwvcHKd+vli+k3Xa1Y0Mt6xpwCdNL+WoM5eetc3rctP8Vwq5ahfnl+PKbvBjmWtKJHpRm688O929YchrbffLNkNUl1U1df5b0lsjlica+1hVzE/4s1ShunWrsU1v4xHLN118dWT//kluogb9WXj/TZMuZ0Bz0f0T+McH2jUShvPsiaf8Wu0qeHe825OdS3BOQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNISQNIRI0gZ+Lpf4P8QVpVWXCYGAmSiNQANJQ0jze5G2pxJo+LU/QkYQxAv+C5yMlUOVpgmYAAAAAElFTkSuQmCC)","metadata":{}},{"cell_type":"markdown","source":"**BEYOND IMPORTANCE SCORES: INTERPRETING TABUKLAR ML BY VISUALIZING FEATURE SEMANTICS 30 N0V 2021**\n\nAs the imperative for model interpretability increases rapidly, [Amirata et al](https://arxiv.org/abs/2111.05898) point out the insufficiency of feature-importance measurement methods which report fature inmportances per sampl or per model. These methods are limited in that they do not account for the relationship betwen features. They introduce Feature Vectors, a global interpretabilty method that discovers the inherent semantic relationshp among features. For the semantic representation, two features are similar if they have similar interactions with the rest of the features in predictinng the outcome so both features look identical to the model. The figure below distinguished between feature importance and the feature semantics alternatve. This kind of visualization could help with feature selection and feature combination/engineering\n\n![Comparing standard feature importance and Feature vector](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZUAAAB8CAMAAACWud33AAABtlBMVEXl7fb////l7vclf7Xl8foAdbD1+PoAAAAffbT5+/yysbHl6PEAd7G7urtlnsXm7ve/v77l5e5qociFsNBRlMDb2NXp8frK3Orm09vB1uba5/Hmx8/l9f7l3+jmw8vt9v/m1t51p8umxNvm2+TnqrE6ibsAaJzmvMSzzODmtLvnoqnnkJbnnKLR0NCQttPy8fHT2uLoeH3nh43nnqTogIWRlJnGzdRHRUZvbm6zub+Eh4taWlyqr7UAYJiPjo7i4uIAbKzocnc4ODgnJSUbGRmhoKD21Nb25Oa9xMtrampUU1VDQUJ3foIzLy8oIiDKQ2eJeH3Fpq1uQUTu0sT5rnnzu5jxdl75n2nzil3AvtW8nJg8HWkhAFdkU4gYEAxyZ5OWkbL1oYWKfYGUWVzGlZvzjWjvZUvukH9sU1WMZ2o5AGRaPn1FHG26t9CpVllDMzOsgIXJa2+oZJiTHXSxeqbLsMqWL3qbInCrX5NQKyq4kraFAGmaS4qvRH+lCma5cprHYYPBHEq3Kl/Qpr3Uj6XLU3W5QXOPYZjYUmR3MICEK3jXMELJfYKegIXm6cGSkGtwbVHk6ti7NCpiAAAgAElEQVR4nO19j3/jxnXncHYkrjSrpca2sIiAARADaxgkZVIUSZESLXmlZVY/d7293F32mvOlqa/XpL02jZt1zknd/GjSa3p3bf/jmwEpkO8NSImrH8v7nEdeWSRI/Jgv3nvf92tACt+M+RvkbZ/ANyNnfIPKPI63jMri2xhL6e9nte/on2fqR/0eju8MfmrP3sqJDcZbR2WJ3OWghBFOHF6WDdmUFZlIj9s85JIL9aOGFKGwhRPEQTloBJUwEjYn6kv0Tk/z/xtUKGVUkJg35CYvc4dI9Vr/pINkPxdjsEkSj28JBR/3CKd3h00OKnurtz5Wlu8UFT3JQSofkQjSV/RqE6xES32YEJuXRVOUiWT6q7c/clBZvS2xqO2vFVZrGxqRu0NF3/i2LNuRbfMBHm+0E4UNd2QlrIrgDpDJQWXltlAp9JcL/dqygmTpblDRmii0k8izBdcK6fp749KO4iiUtwzMdVBZKazuL+/Pgkq/sFbbX165E1T0HNpBJYgpv8E51HvlsRPZ4U3uFY/roFI4rq0ub1wdlI3FtXa73lP7v31UGJXKsFcVIuzG900Zk6QabAlNAG5lXAeV/T1lI/pXR6VQoJ88WHhYuG1UtBVIREOEt8ea9CFssSni2znEdVDRGmxlFg1WoAv37t82KpRJXrVjces2We2fR2FVUeYbP9B1UFl/9OjR0iyg3D4q6h4OvCTmhM00U2MOy2zHY0R6W0E429EuH9dB5eHCg0/oPKFCKQ8r0mFXmiRK6NCRdClX/mKY/vDhe6l7OeGbVqlUskqWNTwmc+wouFnJvA4q79y/tzBHqChNz58on33S9JSImk39Bx367Q6PeIU3xRNlIDZ5kzeULWrKTdkMqzKRjgiHn0Q7LPXrR/VWu3dkXRyY2XxTTj7w7OPmUNkf/rwlVBiLZXW6MbHW1+vaBbSVOagozhzpIBiRhFHwQzgPhR3GduSVnSSwOYfQWEdnfr3Xs86s7C0lpApZ58YU2c2hsryysbFXO347qFDmxd509e7v9Op7n3pO2UtsW4ghFnmWZBAOS7cKaTtJnDhOqE3V4KNWq1X3e/Veyxr/jvJXE/umcLk5VAoHOpAy/cu3g4qaraAip2GiJ1i+/KPk00/DkEy1GcauNT5SxKEWLS1WtLTe9ru+8rvqaB+U2XZk3wguN4fKnvpZHgtw3RUq6oYWT4iYPM9qntJZ/bTdbRlTecVDKMFhDmmKBldOEPt33/33yt7nCBkLlJm6AftyGSpLtUJtf4KxMKz9pTT5FlChgje0WZ6wVZmJWJnzRG2n1jpZL13jSCmfaPIq+e53v/ss/4Ba521e37EcoVJb2SvUNjAqhb3lpY0J/jtE5b2Fhcd3jgrjYRJPVBqMBaQhIiVILL2zhxzsGiMNroX/4T9+j09Sgsrwe2V+TVwyVNaK58VCrbiPUdnYG5/DKag8vH//vTtGRdGpYMvNj0VpO57Y1UDoRKLFX10bkNGe/fp/cuIoFBPuBoVc1bmeHstQOewXikuFk2WESq2/tnq6kq+a3jIqyrqKCpmACRNBkkXcS+KPv/+frdwPXj5K6SDjzPiznk9I4kThJIFhZJOLa8CSoVKsLSlUjvtvbO3vGhUmwyhw87aouRLlIBoxLevV9//4++TNhKVUJ3x9fd3KaEKJnDzXpFiZf9EUQX7YWPHkCudvjMsIlbVCUUnMKkblnXcevj+HqFCiHAk377IVB+ZPuDMewael//LjH0yUlbFUfW4wrNWyzuq9fvZ9/2xn4NZr37Eh4vxcAXWj+I3jlhkq/eJycWP3aQGjcv/+g3fnDxUaiErubGi11gjxdJQsgkFJcWB6H5JLXdYiHeERRzn7IdEeI7sobPFbfb/d9nsXOyjVi+tWthNXxk6UH3hjrCnfUFwyVBQsxeLJfg4q9+cOFcp5NcxTXtRVEmTnhV0u1Je2EZaaRSZEYHtO4kVe5ETK+0vCREakyiukqYzCE1KhCQ20/Bwd9doaFcEu9tQDTj0TsuzlG3caNt7MqSyM+Su1Wvq/uUeF8jjKU17UDavhdOe6tE6IOHI8r+xFthcGnFPKjB9ll4hDIt4kT0i5FPjMYvRPvvcnKSzUOkBOj2Z7TWnn4UJZlLwJLhkqfV3aUuvU/h9AhW/mmViq3ufB5DnQTiDh8Q//NPx8xxlG7CfqfTooOiKcO3LLrkhPfu+/fi/VaFZv2zc/znglNzip3tnks6eRC2PWXo2DlXlHhbqV/MunzWBKMJ1RTiPFAqJey693rzxPaRKG8NBz/uzP/luaS/NPe3m8QZGuyM6TUyrtci4rmTZGqKTqy2TGc4aKG1Tz5oSIJJl452vDnohNkSj10+72utu58zplpFrODiMvDki9PulDrmMn0sSb8jAJZxSXESppcd5TgxnPFSqUbUlpzD2lYVCZpLoU0bJl1dF5fDU1lm9Z6t9sc3RxFKo8lMaf/2hiyRFlntjKcWuZIydHhXJHhspycW915SCPGc8PKiy0HZNrUsGbE3S31j5e7DkXeXzr1Q/EdSIvlNWLigRQOaHkiBEvyPGimIjETBy5MGLGh8XicW2eUaGhHRuzoTy5qp2vuJUN9pyKGBU60vqPf/wX14qHWe0dn0rZkBN8R8KkndgGadcu5SweZWGMGQ+jXfOKCiUNQySUe+1sTXDxqeJFEagstl79+C//kl8DlhLRaWGdB/PihOQGhpUBkmVTjTFFsq9uXEao1NJi+fllxm5i6gblTm7mXqwuCmtIG2+zfvAX//1No5Tp9y9osfZCq46dqzgpjUPTjqh349yoXd7IUFkppmNjTlGhpMLNN1kuS9Y5wWDLzotCmZb+IkWvs40sa1+ZcBb+eTf9fmnwvapMcu8JJUq2wcaoUJ++ohbLUClmqa15RIXZdmBckevkMy83TGzvsgKtARbK0xe29MJYfSVxPMdx1IFCOXD5TXRKR1r9WRaxUjXISBJW8wwMdePAcPY1Pb9ieL8A/ZU5RYUKx8EXz2gjl9gw0hDedCLKqEsEj0U1bAQVGYWxdKQt7TAMA9t27NiJnC07Cp20EmZsT/6x9nSso5Pt1jBAybgdR3myxUjVhMCNrxh/yVDpzLGsMGqELdTd6OW58ow15IScR/q1lCJ7vCmbIuI25Vnka2xoVcaJLSLZCCvaOA33R+tFTRWsk6VWfbs+IA1UR0qdvLoB1ysb7zK+eSUqlqFyXDzpdDrHRoZ4DlBx4wjfYpQnYZ47SaNYTFbeerIjofEYlOZNKSumw7pJm1cHUQGFjNXrKFtf4seLu4u9rjX6KNnMy3Ax1jR8eio8Q+hzRoZK56yjxxyiQmOJdQG1ZY4uoMSu5NNVkjqUJA6rtifILH14GhvONZTKQxkg4Xd2u6Xt8dQmpbKRgwulQQXTRqbM1uUHL1xWD/bWUaFqTgxJiWzTnSTkyaTaEkrdwI4SW6YNjrMO3Q5OZGz/1V+HCk9doLd70oZcjrIkMjmfDs8F6ESZbFwerBxDZX9DjbnzV5ht3m9BzoVpljwBE0akXQm863VGUuYf/CSOnVK/U/e7i4bXQ1kjMMWFORJrLOpWzFsKjQyV/c/m0l9hkY0Fxa2YJJkypzph0hV9lRUirt2JVyq1fBaK6sl3uHvczvFFqdg09acOgeHPyfAy25KhcrC33Cn0twu3gMpGbbnQWlrZq82OCrUDpL6oTPBlpvGMXOWlcFKcLHnThm4w/LMjS+3QP63ycrebm2Nxk7Lp07sVB/n0LLjMnSyM/JXl40LhZHrWS/s0tcyxubKsdPaXlmu6g3JWVCivoPOnofQMYuOWnTz3bBAWCWeIP00ZpQEtJn77oPWT7QrJLaKl1KRdOZaE0ifTYRlDZfWzQmF3etarttxfW54ZlVq/r+RldX9mVJgToduMsi1sPgkLy/nupHS8+MZ6sKydMx0Cs/ptv9sjwoukyHPqRZhj82SMeBclMVbMYGSonK4sFY87Ok88TYPtbexvZDN6ZQ1WWFNSpjPQs6Gi+C/ySZhsGp4LTUSe9WQ8Uuz5UjGhhAFH8qKrxRx1XURhnbXODnol/cHEcXKCk1RBYIYhYuxcMSecFnzJUKntF/Z3T3RCcgoqGxv9jf0sCHBlDWZ9svDJtwqzokLdCN1iLDZojjKdca4HV5Xe9G74gZcY0JhUSUM0+SbfVF5JRVSFgjmttgANXoNosX+86OuyvfQQkbZY5qFDMwrJDMdFU8vJ55ahMljNYG06M9YZmEyBzYDKwr0HM6NCCZYLGjrYA6PCyYmPqxvZy4+yDzZrz5A6av6bsqIcfY/o0jyZ/oTEETpUWbXLgSdEtmaLf6ANvNXdrusc83BmeZKXeaSK9Bnn6VaQwqO8PFlYCjA6eTidGT96/N7jEQ63igoVNpJxRprGPajeMt1JZldFntIfbE2X/AiqdhTanGddeOOLUaVajIvAiZMkTivIaWl9V0NR8rvbx0cj/sVCJ85JhLqeIS00DDGf90wueTEQKk8vQWXh/sL6HaEiIzixzDNK89woL0lOm2JSCdIgjx/FthyEXWhpYk/LIEwpRKzExhF+S1uVEmnVl+qdMVqsM49bplQyZoRTmeMhpFhlYunLEJWTQc6r+HRpOioP7j24G1RYkkDNRB0cDaO8agafFEu2J2CipjmMnS0ZjpaSKq1b6yQrJSqlzd9WCXxHIUMjWSnaClH/pHe63eUW3KsjjeCpfhfTRyawRmYTI5VDVDZWi8srKyur2nDMAyqKfsHUI2VlzC5D6RjT79qTqknSdVd4DBf3slpHR60jMuhOKdX9ulVfP2oj4VE0zf/hWUNEVru1dLx+gD1IJhzHYIGUS8xClLWDTIza4YTKl8KFBtu9StbrzlDB9IuJTWz6SWy4k4xWRW5EVt3xZVEhxhpFdP10qddrW5ZeGslq9epHO6V2y/TbrXbd5U7wN2fhztKZUdBKmRPEhiExM49KqiAKzKnmE7HCTLnIu0KFNuDp0wDXgVFqlFFQ5tl5AUpKOdG2wTUVm25M6XWPrJ9+8bc/o1b9ZNHv+S2zjNjq68QKs8TLz49tExV9iEgYwTnXM5QYfofyJFdYMlSOs4l6+6gowoIwsJFcMMe8ObnjGH6//igPkmTicjsWJ3Xr1S9ev/7yK7bDW37fzxGVlBb7B53ekt/qVHOqVlPVaVR7UNJANwIVZfBlhUrueWWo7BS39/b2dqZnve4GFcob8FSN2DcNJL41qZRmaFBjEgflyWGX1MBT/vqL11/+3LfS5iMTlFL9cOBB7u4e+T1bZ3lzJDInQE+ljW4vjgJ2lG3mRcQyVNJM5GUZ4jtBRXFiJOcxYvbUreL0MFU809TR2v+OLo9OUvLqb38eskkLUVldnXi0jnYW6zvFnqXmupGX8mRh08gExTZSxawpkLTYZkXVjLnIu5EV7sAroRGcAyaNFirXzI2lFRd69TbzAOP7Hqw7oWQqde35ICwGjqdb7jQqva4/qBzXJf45mce8FKmZoHNRXJJVcwJiI1T29852VucAFebACBGVTSQ6No6GKe2R04HoBlE4eXGXQS2LFF4QKT+xov/pYJh4whs8IoPClsF3rbNBtPjseY/4F19mDdvsulOKFRdCU7aFdBbm/G7ZTOJlqGwUn55sF4/fOio0DOBriZUAwbVhVDqBmQQkVTGxG2+wBq6XJLFiCKHgZNSHp6aWxKRCnpAqsdMAgN9PPRhr0eqddzKrw3hOYQvVxffoLQ/TYUTElB9l6LAMlcM99Wv/kgzxHaCi2BWYc2bDyAt1NyWO8wWRYWfdOJiUxldCIIPIKYdpA4VZwDrovSPckw1Z5aHf66W2vnW+214c8zAVEa+a0uJuYX/djVD8lApIKN2q4eIXoL9ySa/X7aPCUI00S2LMJXF20m0Y8SQmY7PQd7CFEl6VDWKzS4srdIgytCP7r3slRq1ea7G+Dd16dWQzs2NmHlmI6LBMEM3HTG0MlRSMp9MzxHcgKzDRRQWOfgl8SdIoNqZKa+Q3VzHqKNPhXD2PrzRb9zBJnNDvkJJV2kUepLJ5ZmJNesiScBtFWvgmVAcxBjdDZbnY6feL54W3iwqLoqnSzbwyNv2m5yLEVm72lrIoiGZdTN2q15kby0r/zPfbO0ZFP1eyjelwEqIP4RQERefsJshJzlAprJ4Un+5d0ut126goEoMuELF7B18wM/KxirbmLSWiNFeSBJPWKZo4FC0ulZTfE/p/en68TY2ov7oFImz0XWcL3TpuhKSlAWCiIbzsOfNXmOch2RaIv8CbjObkMfiTnGw9I7ZdJRMXd5s8rJ2dlBbvnHc2tvJIHaWxNErWFH+D5+lAP5iyMrgwFkHrOUJl+aR4uPN2ZYUhzus24S3GGhLZTSNn70ZejnvHPF652irHePi79dSD7C+2t9u2l7fiKLMTo5rAMZwSRFpwtBWWW2So9Iud5f7h0+l5+1tGhcJ8Nw1xPgLnVLHTphiaUSmeBtojceXuNzCsblolabXqlkV2febJyMw8UreJ7ZhbRS2rNIQBC3TDsQQY/AyVQb/9OV617S5RYTGkkKwCZpgyVLLnlpGRVEo+MXtGpOdMaDLOvjc+wJaLlrvT7friUctSJM7LC4IyB3tMhptlQzVACYx/ucm4bi7Mkb/CoRdMoRmlogqngyY4q2Q4bGlyLJzSYDVYFYwLIcIwCAMphSBkVHNkdZ9rq1Lyl+qd56dpbQvjjuNgkHXmEVkSgQSdBRV4y8EAKwXB5AyVvY76tXH4FvP2zKkiUw8vnUOjqswsDsWYC9kwe3LMWDcKCRkHVadqR0Fie47jeUkcxVG2qLRV1wUtJb592rOs3vlg5TYWhAnWWFTdQiilLbA3idKkLmwAcRtjwpKhcl4snj8tPj047MDnet0ZKpQj/tWA+itswPARLk5QmtsI7/MwZ0mCdIuCKhAVWRExl1o5XnQR632GevljWy8qXX8+qGI9ov3zM98aLkGlE/BGkETX28BDsCqUdYmirg5SBWNGM0Olv9zvp/9Wr4OKfkDmm6IigCZWKgBcg0DhfYHCXEpRG/FaHubE94lWUYJURIXb+eWrg6pKhzfEj3YsHZ7sHvnKhbRG7V0sjIwgNZVVZOUwHUZ1eQwZ/M3RHgs36698svBJziqVV0KFVxErBgLPUKSFhSjyojSA0b6TUyCvv8pima6QMN17UdhYxbpewfLo4Hyn7gMHUrcBGMFPG7m4RnYStjZTAS0LcXJQWdOC0r8mM1bT/+EbokKBMFBYOUlt6C6zGOoLSiI8SW7uUiJUrxHhXe1xKVa7btEwCv6I+d3dYxQCM6tuFGdEEVFFySEMm1C8m+PyRnnVRKVfPDw4ODicXpN/i6ggM4JLj4QNdTR2S3CxCGVR3ppVRDq62vVq/qT/XNFiyoL/8blj+cbCu8rA49InN8a+IpRfCg3+uHSkW7NvF6C/oseVUPEXF28UFQpXZUZhVGpDU+8mOAlTwSU90jNzUjQQTXrlsIvVPVdYWD7tnhz/jczpRZUypvgdVMYWI1sJ2TEpw62eicpM9WBL2oB8+wZRYVVw4zHU/xygi4X3IHNwJIrZZYN7sVBgFTN1WN2eMvCfndUXF9tnP9wkeZlHlEVkASy7U/catB0cWEMqASzMu7jXMlT6xxewXAmVB/fu3yQqFNYZMg/eY4jeUAeHYpAr41aMyjDNkg3vb9oo1XW6vuR3T7Z7ioIpb4oYJWhmSRSpwvsHXQjkLJQ78DIvLitDpXaVNY+uiwqZhAoOUACQEGdWcoWCmCglqO5Is2IuDGZclNPfGTwiyvLrO5/1tYGpmrSLBUa+B0W5A6BraQh0GHOAsGTd0Rkq58fp07SvwMG+9f777xeGqDxYVy8emahY9x4/HsGToqJcmUXyYJIGg+ceNMBLEkNTj/wABwctBa4TpTxnxbex/Q8CL6DoqMSLw6ValLyQNF9PzcXIKA+Q8oTESvu24CscsmcOTp1eVK4XZrYr9J2FB/cuUFlYX3iw8FEOKgv3H3z46OE7DxcvUHl0b2Hhk/UJqFAOBR2eOsoRsTKkLsiH1su7GeU/kZExvNik+76IQxJeJmUlY97gwfe6qnKsp37Y40KJ0UND0cnpPBxUxjBwyZwt8BIEWDXs6chQOckeKKxRWVfW/NEEVBQYj0eoqHfez0NF/fHhwv2FESqP7ynJmoAKQ4HGBrhODi2mh3zDCmRHLMKM2OwcGe5Jr9+qnPxN3QupkHF4zCPR0KtKKof+6ZFZ20oJN5LPyp+E+hVdCxco8AJPD0Yw+IDkFEZ1xntjXqSavoU3ReXhvcf39dsffqh2ckVUQMQUxk+VJwO9fA+5ZsjScw937iW4l2/wQf1k3GBLBqkX/+qVe9FBTIT0nLKixbnL9BlLgBh02K2CM0LsmMEcPcxW0KFgZagcHpwfHh5edHbrWX20pCzB7Kh8671UsQ1RKTx8/PjdxUtQoQ7QOdCDpBJ5Mh60OchfJsYKXLZtupM6FeYkjhiuTcy++sUvfpbtVqNjtXqRJ3JsEQtw5pE5KM4gYCISdRi48A6EniQb3ECF/DiYnuz1dz9ZWLg2Ku/dv//OpahwAANsi2DIhYYeAIEJYe0fI8WP0zL6Pb1mVTj27An/p1988dW4wirVuz4nlcDJaYY1Mo8sRCmIKuCEyCOGGg71TgzV7wCVpdWLURtD5aMH9+/fBSqZkRueKKhOUK+mWEtKtlAoGQctcxJhyjlqclAlzMJf/hQsP61pMU07xHJWUmAeOgjHdNgDdw7MMxJaBp+1x4Vl2GY0QGXorIz8FYjK4jpZX7xFWSHj/BB1r9AATgGKy8KiUqOljRorDqUsWeKwC7MsNjb9GS1mIrY9I/MoJbZluOwOxBBwJgWq4CAALCy92qEGq12MQg4qH72rSO0jnTq5HVTYFvBWGAyvIC8fejIMBi0FznoopouGIuG5LJnKr0ZCOXryDWWB+UwCHc+CsFCYN3Wr4Bpg5JUGQOHBVPEgeDTNrmSofKTdRYXMu+u3IyvgflYuNJhp6EF6cDY2p/kGSjE18HzSvBXf0s/+3ddfZ9bNOqhfLEChy5YkXk6duhXIu2gEzVuAXEng0yCmb4MdpdTk6qiofx9NQOVbyrux3hgVKpswgjfOYFgZ6eTxjYie4ToS5njGkklGjVC26csvv74QPKt7sFjvjzVFGFkBKnGxBOyvVccBtnETSgdwHaHoKPfrplBR0+9fAxWQ/AlAExEDawMxuKQTg6vqsRBFP3A0RMnSxHW52Ku/z7ixf3x2/HxnlOaiboKbY5XbAekwXMOIEphJQU+PBMyZQyOk9egcoMIiwELAyrfUhgEKECymAUx8o3Ydoy+Exfnd+KmTz/jf/1wO2idK663Pzk/qbWuUFWYEP7UKx1PJE3BvCUBZWJwAk/4E7KkyfvOkrSFzgAqB1He8AkcnDwFG0E32gAXiMBugmA+Bgxt12vpjeiERGTvJr379mxdhFCqS65/ttM/P+Wmxw/2RdUF181QESDpQYhE6+KBhDV0VvA/1VcwBKgKmJKD9fgL0M/Q2QxhotlEZHIqH6X5LAxTKZBBHjheE4rdff/0Pri3KohkW6y7Z3Vmst9rnXZ9eHA3RYZz/hNUgNAQOPysDogWCMFBnp8n9t48KteFUg4tDrYew/EBAQoZMfQWBkphrCipMynZCeKq42M9++w+/1YEwUf9UVKiSkpJV6j0/uxAXXc+FPFboeMAWLuYBX1jEUHTAjmCARsyDXaFAD0FOiSuPQZsHBWXHqNqK8oCAD0sjREn1Sur2aLkd91cvfq2Rtc6UUg0qafO94senO0PzQl1UNyiBOGCQQPQNpYZZBVwloNnaM5gDVIADxiJgVqA5j2B+FRIwSK8YLCfTwSsMCmsGoERCofJCTayixb5e+yUZrC9l1U8Ohr6LosNQvQYo7Q6jEOCBMSxGYSTozyDePQeogJIEDrQScopBCzSKWsKqZFxOxnEhvV5CHRXg01+9+J1+IOTgwYVq46ALtmQd9YYJMAZXOaAcKiJUOQEnHrJH4FlRuwxeVdhUVO7fU6jcG6JyT/0xjsqDQYY4Q+WdMVQeZPkVjcq3DVQKY6iA9YlhCBVeJw5iQoyg2XAj6HxyVCPERJOgHKIvfv3ihaAl0h5uUExpEAfwe8UhLC7sRUO1EpDiUlDMrkPX44dLwLERunl2pb+/v9/++IOPj779wQcf1NS/b6s/Pj7Sfxypt9sP1V/76o+P1B8f6w9+9L76VX/vgw8e67fffzx8+3399r569V7948H3P1bf/+A9tfv9cVRicKMABQaYMAVJPRpuQlsLAITLqNEQLW+h9Anusgh/+tWLFy+4osXZokeUDh8+4HfPB7bFaGAEdwbyFRFTgWcEKgsoTH1tihxU9EMMNtbW1vS/wa+18VeXvb1x6duDhyRcoOJBHIDpB3MHW+8RRrA7hKFSJLTECA2MDnfrl1988eLF7yhdz4ooSOq1ph+0rN1hDDmB0hFBQW+CjcC0sTLwFSFDA3kx5bHkoHK3YwlMECxFQKYfyDnbgnkJcLeFwMs3ysmMdbt03uv13/3+dyGzuqCgmIXVVIlZrdOhBOHQG9gx0rgwFgZjFvCqAWKRMxsqG2kOpobe7a/kfviKqIBuWgoLRKA7DeUcJpY2wd0Gay05KufPqau0wn98/fvf//5/cgs9o5gOw45DDlCiP3kJA40jKbQs9vLl2LeRYbHH49dwBVrYBqJ85VlR2a/VMCiF7b3roOKMTwKPwHSWgWADwREwSA8j5zBiS6ASF2Y0jNJ/+qc/aFQ8XVsMNwVpnqE06Fu1to9+VKd6nWm9KJX6T8S+Mjnqr5JeQf/oJRt79iEFRojDuAokyiBMZkezopJCsnh2frpaKNQ6Bwd7S4V+8en5bkEv5r5xrtfb7xyuFfYODpYv29lg1GCKF5gOUOFGgXcAg/go2gKnA9WLsZxyMen88jevf/Obf0LCHUIAAASXSURBVP7n5KTnI8iGyw4rHeaT0vrpUiz9VqfnH/V2e0cnR+Lzs07d7/d32/Xt3W77f/m94ywFwGAoYhMEXSrgDGDSzH4jVJ7ubiwXVwsbexsbB51C7aCzsZY+7XujWCgUi8urtZPTjZXRAyinjiVoOsBNEwJbOtX0E6CmoUmGut9YnJLJikz4H/7wh69/K8SeE+NWJOalsuWfKx1m9Q+61V5v6aze21lSCB74n8v13cXtOjlZbB35vZcrndG6SLB4GsVVwEkBokzF1qyoqHG8op/snT5AZ39/pTjUYBkq/cH/C6vnV9rlM+igjGMEK78QbbFhzMKeeP1o3QdMz/SSSTZlLPzi9Wvin3V9T+Kl++L0JrfaioaVLHL8stVqnbV7R/6ZQsp/2WntLu76/q7f6vq9z/931xply0CYGea64DOYUN2LNysq+7X92l7x8Plh8bPC/qEGCaOymnYoPX/+1GAFuaMGyQgkxnAbwGELWHsYXtmE9yiQMRQWYU6ckiMqv/zyS0GKdcqMhw8MA53+8yNrveR3f/RXR4u+3+v6nXX/fHE3YidDVJSsLH/e8rPHT8MQN15gB1wNqCyk5VlR0TO9c6AfzLZW2D2uFfZzUTkcfOAq4xnUWYBaRlBnAfMATA4HJbDIrkJvE4VFgnDo3Vlf/fJnfiulxS5ao4wGZMCOz/z67u6xkMedY65lZd3f9nt/vtPxj331X/u0212OO7tnmRcKIinMG0/gwTwYipLNzIw1KsvFwStt0Pvq7xONyrnSXHtDVNauJibpWAQzX4ZnN76NA93LoeKDVX4VgBgwuRI4esQd9Rkxxkr19WG9URMimRYW0Ppzn+pOliZXvGvwMGKf+JWSr5ej9PWTpy0rAs+fhokwmIaAbAXcjcmboFLYPuwvdzoKhOU9rcH6xZ1++uKzISqFTrG/fLZ7pV0+AxYdLBxPq+MTCC0irOSbEplBgWaYVCNbYNX8jBajBWWGRThWvZ6iA7sbAnCS8ZhNG+8/xWEfmPqmDuQ1s6Gyvzfopu9vb+/tK9k4PdvXcrK8pwSlf7qjX+j39QJKp2cb0/aUDSgrFRjNGr96AVVdPFHxQfNOIS2uoEju+Kuhp5h+CXhDwz1aZ2mFBdw/7NuADyhgwBeDjBs6/qDhkM1o7W9hPAP2fdzRoLwCRB6W60BVBwVu/OrRWgOQnoFq/xLZLY1ILVx0IYXW6p3o7gmkE2E8zoEKeWwbDHnDpi9Y/j6rXbmFsQTuZZjtG+f0SDpQggLcoVWgfiAfezLZ2/Rbo7gkDaFkauVTIufPdTyZ2lVgqMe1J0wowFJWSNjheQGNOQeoPAMqBRQcSyArEuIA6T5wRcdlRS+5A2QFmCMQtfw3EC2OYYxe/7baw0/AuYaxa6hLgZyOx4GGDCI7k/K8ycrYeZPNcVhA6h1ZdIAK9QAQlXERAI2IKDIjxj/4f/7lX/9t7JPgCa58cwDLIPUCujYpILw0AQo5BsII3XmwCS5aE7x1VOCYwqiXpmyr5SzpczGeTdnl2NeWCvtXp/NwL5PPBO4RnsklR5snVL4ZF+MbVOZx/F/DSI9GlB/TUQAAAABJRU5ErkJggg==)","metadata":{}},{"cell_type":"markdown","source":"## 3: TABULAR SELF SUPERVISED LEARNING SOTA","metadata":{}},{"cell_type":"markdown","source":"**SCARF: SELF SUPERVISED CONTRASTIVE LEARNING USING RANDOM FEATURE CORRUPTION. 15 Mar 2022**\n\n[Dara et al](https://arxiv.org/abs/2106.15147) propose a novel pipeline for self supervised learning on real world datasets for a robust representation of the dataset. Their approach outperforms Label smoothing, Dropout, Mixup, Autoencoders, and their corruption strategy eclipses mean corruption, additive gaussian noise, joint sampling, missing feature corruption and feature droupout. Their architecture is quite simple. They take a d dimensional feature vector and corrupt it by first randomly selecting half of the feature vector to be corrupted and then replace one of the features with a random sample from its marginal distribution over the dataset. These corrupted version is passed through an autoencoder. The autoencoder projects the inputs into some higher dimensional space before a decoder learns to produce the original feature vector. During supervised fine tuning, the decoder is removed and replaced with a task specific head, like a classification head for binary classification.\n\n![the major parts of the architecture can be found here](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQUAAADBCAMAAADxRlW1AAAB41BMVEX////q0dz/0o6astj/iIj/lf8AAACFhYXExMS8vLx4eHjl5eXl8OH//2FqqE/8/PzNzc2mpqavr6/w8PD/1pCampq41avT09O3t7fb29uFuG//AACZgmF+XH7v7+/29vbDn2acfEiNjY2ZcplXV1eKgHGKXV2Zdnb8zofl5UqxsSyfbW2kpD21tb7Oztagm5VqamqgZmb/KipQUFAtLS1oaGj/5ORMTEwmJiY2NjZ6iXI4dh1+g3xpbmjG0MIbGxs4UC5AQECQmY3aw819cHakkppUkThvoFkfHx/ErbfKtL5tYWaMfYNDZjRKYUGLo8mInX5UhT4waBa2v7OcERGZAAC9AADp1dVOakF0GhrOAACryJ5We0a5v7ctQyTVaGhaMDBrk1pRV0/E5bXT3dBcbopNWWxyiKpEbjIrcABFUkHx/+lXY1Lnh+fhAABzfG4NLgAcTAAqSxlgzSwtOSnvdnYjPxWjUVG8m2m5V1d3Pz8pXwY8Qjvnv4EyXRxAlhFFoxNpUVFbbVPLyzijo7A8RVIAIgAOPADW1lAwMACHh12MjEuQkDVAIiKVczt2Vx+SrYUAMgC8YryBTIGCcFVkUjeLeWAcEBxjoEluQG4YJRKuWq5ZPll/Wn8AFQAcJjR0DQ0oPhDgAAAYN0lEQVR4nO1di2Pbxnk/Ww1wwCUBDvAZuCwF7BuSpl26oCNZko5lhhIZSyIdL/HaJWuahxtZzmtp1NRJXdtp0zjZ1u7Rpsu6Zuvyp+4OfADEmyJIRa5+siw+ABD343ff6+67AyCE1gYq1VwAHOACU3P5Ixe4/I+LXS14ApAHgNfwtHpTBU0P2G1Pxs0GBbJXp/xAh1+lgfgDExBxCtPEeeL36ECwACVPaTPLc1qwXbM8FXZwEzFf6cqm5+tttQFATYXMAshpNFy7QXysEheochNKkMkqAJIt2WrDrumtptWG3Tbtqt5ht2weaB3QgA2dKPx/twWJpUoqbgOiogZiDdhRWcP1gGk4dV8BDLfbti17FiG6BlRMQR1sqX7L87QtoDLi6bJttyEhuGEekAXLQtU2sBxY3XNUHbOGYjS6XBYc2beaALWAT33PVBtUbfAvW203NK+lNFyXHwe6ROdnyhRoTWD4jD/ETdD0SUNntlKDGOuqEKCD4J0LpNLmHQSQVXQhVcUHO/H8V4AFTavqSs4Bz7v07EZVt3CEceniMQsArF85/B5x+Ni4/0izQBWBhc3cj989X8XdHBbIDwQW/iL3nzzaLJwWOKB9DPHu60daO1bEwus/OWYBgJcuVeW4cbjIsiy6yriO/KgSFl58qBpZcBU2HPa3BfrDIQtc/hVAfmt9ff2thbXjrbcqkAXNGPY3T8xge8jg4hcuBL5fYGFZ2Hhz4UtobBijYExET1n00oUg1bDw3jOLyoIxTKMgwI560PCmLCpiYWG9IG9nksAxXLI4VMTCS08tJgtqamcI0V8uDeSHAgtrxwVlgRSQwKWBLnqLedh4WmBhFg7gLyBCJt2d9kdN3exu9zZPbG72N8XvzokoN71FbzEPGw8KPLHoZfafnlsW7PsvTLwBNm7pZrs3GAwH28NBc9Db7NVaERa2l2kxRyyky4JixJHpxPx4X46/ZD76VzE8OiPVysb5yeUm5mGzP+z1uNckMBR/o31imaF7HgtnH4vj86zL3EnKgvLTr8Ux8zHKxYtjFtBMa7NQoYueQE6PoGdOnQxwavz35KmzWZe5cScRWbMECV/7RvR95fr1MQvOlIXNQBVsbo4ez2KZssAucdxJ/QT82LjtZ25P6Lid1SVS9MJ3kiy8H33fvnAl0SPq/f626A3bw51+v9nfiTKxsDXPwcZDAvFPQCKge+TUuOnSB1NhSKEL2fy/G9ejAmtwBeD8LMnCy9b4AJu3X7myMWGBTPVCTWiDrX5/0NvhjwYREvpm1U2PYMQCcc0oXCL5EDw8ZuED6fbJCQuPAMdEHJRaArquW7jTUvVLf1hX7AmUbrtL8FQt3Hz//clDdXSQUmu3Gfn5uxNS3Ikw9Ld3tjd3+js7/e1NEVVGWEho34qgSc12cz1g4ZY3i3atKTXOTGShc2rKwlml1hCQBYgAllttD955+i17aknsrtQmvQgJr0wePzo6yG526ozcmbIAlFT9uBN5PFgSCQhLXoc+kdojsK8DY6IWbkshC7etxHUoceNjU6JHfCdk4WdTFiaKQZhc5cMPQy2Dc8MI7jMtZRgUkboHcdcF7FWBNO34yLTtZ6aPTj6WoanPvxZ7I6IWXrl5M6EYOJRffCjZ4bPskJLrC38JiSfK6r7wxMSlWaYH/XAoAVMShGJIRSIHHXoLL7/yylQvfC0idMqLL1qRwWFHzRSH4YyoKpoGxE9WfEWBJd62+K/Lf/jjpPwCS66rEXuX6S9wbyENZ9OHRK8/GLOUw9Ay/DTiPkU8hsBrgp3Qn7TkNO2wMySz1gFjFaqywRoZg7P8otg2KIGYcQ1EGP+NH6HW1VlmMn1HfCYVZ9M9hhvXY7IweDQVv4ywEHjQZjvy2QgP+zMSsT2UE185xrrhE19lGb3EhtSChk45AaahEAJnWYD+FksIRyYLmpOOkrJQDPu1UTTFutFLmjombDhWEsMUWRZwyyZlXWU2IFcaLZYWolcUUyZkoRgI41FkbUmJRjkjGioNHTTba5EMUyM/I7Cwj34AWQih1RIuERQdY7O6BJNpNLs4O3tZUQ766sZC3xuux++Q8RCin3HbmsAcV3dwt4lzHfCK8o6LZuLRlh17pcfVQvqx8KNzHGV7ICJdzy5yN4pZUMp83hvnUmQBzyMfvj/7HA2z1ILyg9OnT/+oVA/kzmFDKSE2hSx8+1sPlKDhXMpdffyrb85Dg9KZVV1GVt45YOF0MQtUrjdguZ5TxAIn4YFvFtNwNSkLH/+Knxm8ahgjiyf+13hfz1DUTn32NrLc5lIsWHJHLT/IaWGBzDS3IIEjw28OkdQL+FfixIBAg3swBnNdgiHTLWboDCs4jQq5Fv3qsmYdFvcIXe2oGa7GgYA/fpw3ZTfHyIxwfiPBwsffCs7kX6iBACSKRRXIXVrdsiCkBk0lXpeoOpYA89raXnpOwQ604ztZt8KdQ7nywQvBwre5JKMpEaPvq9hWCRb+WjzgRsp0+XdDXdcyXdc1Tc1m6bxq7e4owKLX1tbWLvtpx9hXXuNIdXE0xauzZQThYxYUnSII+I9GMaKIAqmIb23KQgrcjFt1pa3AZsLdtQDXUsiyXxOqLMmCa3vtLOdwUYxZEFKMZEBMDA1Z5tancNp4yAIcxbgi4nULJchiLRPgvbUxriXJti8IFmJOv2nU8pzDRTFhwYaQK3vFgAYzFK591aITQxYwthkk2GDAYIWnCchTEtbWdhNjUkkWip3DRTFmYX6ELCgIQwUTGwNilGHBv7wWwV5c78Z6hEPatULncFFMWHAnahFEHuQhZIF3AwqQy3UB9YvjNvPa2izipkK5c51jZCMQ2yrlHC6KMQuU2g6jlo4si9JSLmGqdizuuTROQsJUKE+LZMA5kTns+KuY9QRCvUCxbkFFhi4jsFTEG9WOjqNzAkzuOGoFty2Mg9AKu+Lf3u7u5Wu7cVMxYuEjeWsO53BRTFhAhkIpZjowFCUpCwglemZEL+jMMjAhWJeFiszFXcHA7jXxc2337t7lvbuChcvRTjFi4ZkqncNCjFlw8z90/UqidSELNsIEcRogta0iFrTGZc4CF4Jr1wQRd/eELOzNDnZPesQKMWMjtNjfKVIqAyKyoAkrZnJv09SKSxXVy7NKgbMym3uIasdVYcyCY2ouV/TUNRzsoETAuv9sIu6MaMf5lDjZm1WOuzExVF69yHEosmAzw2CKing8pBCSSAKnzO6K2QgRUTsiuC5WafZulISEE61cFIOKh8ICD44tGxHuO1qGraK4lkjJO0ZsBOMOp27bwodkVjy1lgIa0nA5OfgyYuGZ1RkIENMLdDyzKWEqU/KOIQsEK8yG2MGmZWNWZGZdKJOp53Q5xdccy4LaXerkv1mU86A/+TSZC5iy4HCdaGlcmLhqQY6RfyW50+yMTMVa3DiMMekRqNZY9szgMbzf/NMDDzzw+D8XfVxyRk/z8eDM90K3exz0FhgJLLWC71iYit1UubHvXLp0KbARVlvOULyGSMScKz3MQoLDs/OO/yLyZo8XeiifJuqm3F+LM38zZ7yveepkWJHsxY3DGFF/Qemk3zo+XS5DO0ZRrYzGaXg8K1cSIqViSNAwLwmwE9F5WZV9s14T7qQJTMUscBpKkBCb3TWC++t5SVC9Mp5FzHfU5HpSZqpmAWhlVHHqbHBzPhKoVMKIghQP2vFq8U+qnIVSeOOj+ccpMZ6JL1m3ZKpE+flzHDMeNO3GBi/w3wqUvicmji6vTLNwkHHKjTcj3xXaKn0B+8oFjlgL9a2ZF7AkUCq9F9yK8MkvLsxC6jhlAa5uhHeJ6+V7zyjjlphyYWxFPJF5WUid6Tc3DlIl8uI0FjC783CYmoMWYK1pF6uQBfxEHJmJopQqEefmd2KYrQzg56yPHtiduXzhTBaAO/Wqq2NBO5uY4ZZZGZAiC3ZBZQDYuD84hztKZW92fOGMURmBwKtGVbJgnZlMcgwrA7Ls+VMvJmShqDIArG8If1NPzmoqgMWeEFOPxcOUrLhVZxIGeF+g9FwruURlwAcfFFcGJFceSKsM0GbPeaeso5QBF0OLWlCn/A8Fhk4tSyxMVGvrRjBprZSuEk0azXGLywIV0dMj0znxZ/IqA6ihiSqRqCxgvaAyAImZnC/tryNpoQlcJsQ20RVdoTYzANQVZlli/L8uqUGz3nFQIRypI9MRC2z2cIdJNds8G1YGTFn4XAsOoLoAtBXFVlin7sH9Z9eNmTnxjEzVwvuRyoDxEURq1eznfrHfXGxMSYM6hIgiAyJdBwhBG40Nrhw06w0mF4JJTamxHhx+dfZwVmtutfzbk24QrQzQg/cDBIUBhsHqdd+6cX2d39AYerPexb0pCTcjlQHjg7DU9vU7H360EAchkkOUuHyPaHBZTu8RhooyKgOSioGKiRr7M+s12TOVAT97OVEZ4Ihe9Myri+aRTYqEXnTEjG9zNEecjhQlflbgvJjrKYITMdmCvydCInFsMPfCdbVproMFh+dXBkxLZbIrAxI56FAt/PRfX5myEK0MAM9fXJQFCrEiK8AWY+kEGgxjYCuGaNrUUnK9Y9i2zXUoNmweulg6UrDNj+ZSrODJKBsTocnP0+ZBn40UBISVARkew3587a7QW3j55s3UygCwfmXRFW0cGxlEB4aLOQs2xrqsQWOWBSSKQi2dtx8rNiBQ5q8QRAjG1ABMn7Dw+pNPPvl6CgvW7dTKgA/S9VleZcDLL6d7DDPRVMVI8ZpGXuWM90vhOJOYyQL5t7Np+Pd0j6HxRMxrGn4jFdFqaRtXmEyPeR1TFibZzvz1OzNZmA8bYJnljlnQDGpQy0DUhrEWTFngFot7EwbUDMJ9K663Fe5lJdLgG9WwkFJbuwLoDgMM6LqtK7Gk25QFiyL+JoRAt2xmi+mdNlZoPIyX3xA43MqAgwOa0ASmRbk4xHJNExZME1iO6TjAMh3eeMfUHUux4r0Dv1nJnPgDVIlUiUQskhtTJiOXiioDDkkWMhH2CIh4VKRrLvetuSzwP9x1QiJgcFE4J6MiFg5ZFhKIsCDzSMvi/pWKHdWxoI2Zzf+zDBuTqln4ysoCRdxI6LrBjQN1bUpdhBXoKIRSXbGnyoRcEVg8+3qATPwygS/wVmWvyAtjJqUiFipYu6tSjMrByo/WVlQ9doBM/DJxOCykVQwdJg6HhYOMyiwTgRv05qpZ+KrpBeO8QHkWgsMXbsL6xpFeCfkYx7in4NppA6gwnJIGuZOo2DaKTBPRbWXyzC03e+Srji25aTiOw4MkKxhRotxTNhFGwVOTh1BtBDSJMYtpiGogGLSwG21mAculVHNHL1PnUHYCqQx+l2gy5lFTx5ex30USwwZRPMbbSfw26siSw1kgBEmW5PvQa4pUH1Q1CUhUUj03eNlueqmFi0cFGjSJJ9ZfsRnoNGpd3NFc3wOe7XnIq3VZG7QEC4bBWWg4NbnbJJLEWXAFCzWt5QYvq5Z+pFkAUq0tY69L7JaqEpUhyQRdDzQN5rPgKQlkAQBH0j2zq3jEchyg+EBiktXVJDd42fYac04h+KrBovzXcQyVigovIPo+1w4utYKnLkVakInXeGAtkixBRGkiriD4MfwN8TJCBj7iLIwBF/H+XH9V06qPcYxjHOMYxzjSkJm6gvr1SuBMp8tWHQJpG29uLHUthwphGYxRBg1Kqt5RU1u/sn5kWLB9Zui+ahsFNXlzQ7vx6qdHZcdWpGNo+9BgVtXJb+3Fh64fFRYElrM/qvbeufeOSo9YHg5TO3IJJ+OZt4QY8BBzX1w7Xj0EFpDBesP+dmRl/M3tfq8npy71tnxonz7341XrBYv14jttTbno91jOXMBgGT5c/WJd2v6zszbClt2pHaJIwaNRfzR90ZkmwOmBpi5quFewgcZ2j2RJ5xunxZ6S/1H55kNcFn4bZQGqluFTWcGyyRiEEoGGqsvQI4yqVCwhQD3mytCSLdI4gNXWSK9wayEuEUOW7s++FVSCLoGFJ3/yepQFg7fTt3BTZa5HbOhR1gAWrhHH7wJPRjVAiayqTR17/gFyXXBQgoOAh17qqMnSWPj0yRlZcGst3cd+E9coa2LOAlZ9ptYM4hPfkJ2ABdvHTPUb3tyyoJbaTGbcL9LSoktj4fmHLq1KO/olBWEsDimLwi+NhdX5jvORwJGMmcYsVL6EmXZr49Zq/AUj3CZk2OwNTuyIfYV2Nof1vnh4orfTH70UkYZEuNBaF7havSyszHeMbJvjfffEYHswqG/3Bjse956G/V7X40+Gw0EzKgzD+J2tB3OXL1TPwq13ViMLdmTHmEFv0Bvyf82d7pA3nbPQbA8CFma2GDqxGbdC6/cviYVLVekFTctdy53NNG/8M9lyS/SE4Fls3634IkDlWNAfScF6nsPJbUQWC1//yxRkT2V64e8Evsz6sBkWyuJgLMhpZU+P5a498tLzn6SzQH93Xwo+y7zS9/5G4PtlWCjtNsQ5L6cXfp9aA5e3yne2pdxII+G+LzIFq4AFPbJtSo2rhuF2b3tQFFHEm9sN1kI4n8+CGVYCngr5OPX7PBb+mCULw1QW3v7hAVmI2ojaZn97sDPgoXQ+C4ltt54K6mGfzvcX7Glx9EnpP0NCzuQs+JCIKaf4LGx5lIa0DR7cMizQcHepfm972N8c7gzzZaGfaG0RC554R55WiD8c7qZ38mSGYtA6YknpG0kWmAjgUagWpCLF0KjzKxWxAMgcUUTQH0KtoKlqlAUGlXRIW1sq/TwsEO5EFcN5Z3QQnoKI0vF6u9XEN67/1nFdV1TVi432RF14rSV1cUQtSJ/9V0jJFyQ4d1Sqrgo0vI7flGovfI+biO99mSN3eC4a+lHPsUGiLAyyiumldqtLpmohWjMvJMNSR9XyAcTueGKXQGNrK9hZ8PpWp9Vsdj3f90WjZLlbb3fIIMLC21/8d9g91oMdBgWCUnqxSrJf32Jo+PcC/bzlVpQCPRDFzO7myCZuhIXMHtEhDoDhmgmRmnmhGFLFVOuKq31yQ4p7OrJvRdUC7xG/C1m4L2Vrg2ACZv+7Atu52TC3bGzd91Md2jLaUY60/IOozbyds3LdpT+8m6Ydo96CdF9EFjI9huGIhYI0nOXnbcE35SBjQcE/Pi+wn2spP49+/zMeQ3axW5aNwBHD8PZ9b0eeZXkMYxakupE/+otYL3dLxu1e9oYYzwTr6lzMkwUzdd2AkWLIPImzkCoLvaSrMOEkw2MYs4CQLGV9lRPorDfcScs27Ax7cp4wlWBBSV83QCDbY9CuP/hcGgufZbKQ6jGAKQvik2BNIgUz6BxIZLFf7XaEjQFRCs4qwQJ5OBO/z5wBrv3pDSmFBed/vp6F/83YSFUd2c5ROxwi1YozQq5LIVQnNOwUB8wlWDgQtA2wpCyL1ZDkUsMnk0zcsDjCXxoL6z9c2vwFzai3C1RlgHGEUWIubhkbcRBod55O1Y4VoYSq5B0joCGRWEpBqWjqAMjSjtVBqMqCT0CChn6J7rM0Fj7Z/9OyM/EukZr5N25xV6qXe8QIR1cWAlh+vqpU+om8UhqWxkJKZL0UaEY7z6vEgzL7hCyNhUsP3lnVCF2uqiw19LvEHrEyFjhgs5M+OcEutWVMfVljU7fOr2iEbgwz1aske7tlChxHOejXjvAIXYixqnTCz/X31tb2SkxiXNrY1K13DmOmn1Fv2Uwax06aN9o4oriob2ksnPvRuUOZ9clVpTfa6RFNNxHxiu5kaSx89IOPDmfuqy5W1DF4/4hsv3atwH9c2pj11UOZ78jhIGRZenxzofyE3a1gXZ2Ne0I7RpHYaCp3stR4wd7KCzq0q+evHmKViBojoZypuLcwNg4xGo52/ffcQHeTHASm4qiUMFUBK7k35xh3j/bSEPNA+78sEtbWuod9c6uDkyULf15dIlU5cvV4byyJUB5JQ5mxD929jbjTtBbfpHQWg2CXlpsr3b52FZh1oLkLndvCL/9C4B9XtJXzCjG7l/Xd/IHKF0YsrHSzzqUDNpWZfc0LjcOYhavqMlC8+0ZlIFGozZak6uPdOcsYhzELxNKzAFeHjBlmc4O1O3VGJ6aihHF44V7UC3CSmBemIs84UGwEaakX/kHg+/cWCyGU3Xzj4I8CzQailCJYeZblq4LEkv6zwKPeMlIc6J5lIR/uOC87YoH+mbIwwTELAscsCPx564UJvMDZku9VS3mMYxzjGMc4xjGOUQ0m+UakQARQRs1HdI0zt4x/NT0mWDNaKbl6FhTJQSeSDjApxsZkvmL0Gto0ceLM5e5Zkxxs0Aqq/D+8AQNrSzAzsAAAAABJRU5ErkJggg==)\n\n","metadata":{}},{"cell_type":"markdown","source":"**BEYOND IMAGES: LABEL NOISE TRANSITION MATRIX ESTIMATION FOR TASKS WITH LOWER-QUALITY FEAURES 17JUN 2022**\n\nLabel noise can make predictive modelling much more dfficult. Previous methods presuppose that features are of high quality and can thus be trusted to weed out label noise based on their marginal distribution. With tabular data, things get quite complicated as we do not know aforehand if these features are truly informative or informative enough to be included in our label noise treatment. [Zhaowei et al](https://arxiv.org/abs/2202.01273). They divide the input features into several exclusively uncorrelated parts and down weight the less informative ones by the f-mutual information between each part and the noisy labels. They show that calculating the total-variation-based mutual information with noisy labels instead of clean ones preserves the same order as using clean labels. Thieir approach provides a more accurate, feature-quality agnostic estimate of the transition matrix, outperforming prior methods on datasets with lower-quality and on UCI datasets with tabular data. Their method requires no preliminary learning, and is based on a data centric approach proposed by [Zhu et al](http://proceedings.mlr.press/v139/zhu21e/zhu21e.pdf)\n\n\n","metadata":{}},{"cell_type":"markdown","source":"**MISSING DATA IMPUTATION TECHNIQUES**\n\nIn this section under self-supervised learning, we examine state of the art data imputation techniques on tabular data. [Lev T and Simone S](https://arxiv.org/abs/2210.10446) seek to make GNN based tabular data imputation more effecient by extending operations to per batch computations as opposed to a full run on the complete training set which is an improvement on [GINN](https://arxiv.org/abs/1810.01860)\nTheir novel EGG-GAE module sampling scheme enables the latent graph structure to be learned end-to-end in a fully differentiable fashion which helps improve imputation quality across a good number of tasks.","metadata":{}},{"cell_type":"markdown","source":"## WHAT'S UP WITH KAGGLERS?","metadata":{}},{"cell_type":"markdown","source":"For tabular data, what new discoveries have we made as a community? This question isn't tenable to absolute answers because tabular data modelling can be very niche, so what works for one problem may not work for another. Where to start? We shall discuss the journey so far(over the last two years) under two categories: predictive modelling techniques and memory management sorcery. The latter category is a pain for many kagglers because time limits could mean having to sacrifice some training data or complex, albeit intended engineering scheme to meet run time requirements. It's that important. There's one paper i'm saving for the end of this essay because it has been extensively used in the [ICR age related condition competition](https://www.kaggle.com/competitions/icr-identify-age-related-conditions/). It seems to perform well on this problem and almost every public notebook uses the model in one way or the other. So, predictive modelling it is, for now.","metadata":{}},{"cell_type":"markdown","source":"### Predictive modelling-what the community has learned","metadata":{}},{"cell_type":"markdown","source":"#### Reimagining 1d-cnn for tabular data\n\n\nI know this doesn't fall into the last two years bracket stipulated in the competition prompt. This idea is actually three years old but Kagglers cannot get enough of it. I have searched through no less than 50 competitions hosted within the last two years, and I have come across a reference to it more often than not. TMP in [this](https://www.kaggle.com/c/lish-moa/discussion/202256) discussion exposes us to a scheme for implementing 1d-cnns that overcomes the irregular feature arrangements for tabular data. Usually, cnns work with structured data, that is, data with inherent order. Unfortunately, features in tabular data have no such order. The idea is to pass this unstructured feature set through a dense layer with units greater than the length of the feature set. Next, the output from the dense layer is reshaped. When the 1d cnns are used further down the network, the goal is to learn relationships existing in the transformed, now ordered feature space. The network learns to rearrange the feature space so that a cnn can be effectively used to learn relationships in the data. The dense layer's units are usually above twice the feature set size to encourage more detailed representation and ensure meaningful rearrangements in preparation for the 1d-cnns. ","metadata":{}},{"cell_type":"markdown","source":"### Time aware features +1dcnns are all the rage","metadata":{}},{"cell_type":"markdown","source":"I participated in the [jo wilder game based learning challenge](https://www.kaggle.com/c/predict-student-performance-from-game-play). The first placed solution makes use of this aporoach. First, embeddings are generated for categorical variables. The time dimension is flattened and passed through a CNN layer. Finally, the output of the CNN is multiplied with the embeddings to create the time aware events.\nIn this same competition, [the 1st place solution](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420217) dumped transformers for a 1dcnn approach to modelling an aspect of their solutions. Again, [someone else](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/398565) used a  1dcnn with just four features from the data. The approach acheived record inference time and was placed in the top 10% at the end of the competition. So, if one has sequential data, try some 1dcnn. Turns out that transformers need larger data to train when compared with their 1dcnn counterpart. ","metadata":{}},{"cell_type":"markdown","source":"### Graphs: Did I misss something?","metadata":{}},{"cell_type":"markdown","source":"Yes, I did. A susbstantial part of this exposition discusses a number of graph based approaches to tabular data modelling. Truth be told, not many kagglers go this route but I stumbled on an approach that used a graph neural net to generate embeddings for a portion of his data. When asked, what boost this consideration gave him, he replied with a ['+0.0002'](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/324070). That margin is hardly enough to separate the first hundred solutions on a typical Kaggle leadeboard. I hope that Kagglers start to leverage the capavilities of graph based learning for tabular data. If accessibility overheads have hitherto hindered its adoption, the authors of [Graphnet](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/324070) have made things much easier for seamless adoption. In their words, 'For AI to achieve human-like abilities it must apply Combinatorial Generalisation (CG), and to realize this objective it must bias its learning towards structured representations and computations.\n\nCombinatorial Generalisation is defined:\n\n“as constructing new inferences, predictions, and behaviors from known building blocks”.'","metadata":{}},{"cell_type":"markdown","source":"#### KNOWLEDGE DISTILLATION ROCKS\n\n@cdeotte gives a detailed rundown on how his knowledge distillation [approach](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641) placed him at 14th in a competition with almost 5000 participants. The idea is simple, pretrain on soft labels(prediction probabilities generated by another learner), then fine tune on hard labels(0s or 1s). There was another competition that deployed a similar approach, this time, it was a first placed solution. Seems like there's more to unpack with knowledge distillation. This image from @cdeotte should put things into perspective\n\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2022/summary.png)","metadata":{}},{"cell_type":"markdown","source":"#### TIME SERIES MODELLING TIPS\n\n1.Someone dropped a hint on what models would be most likely to win in a given time series competition. His first on the list was an ensemble of NNs, and one of these NNs was an autoencoder. The writer believed that autoencoders and NN based approaches were good at extrapolating time series data. He was right. In the [Jane Street Market prediction](https://www.kaggle.com/competitions/jane-street-market-prediction/), the winners deployed a supervised autoencoder to add features to their model. They won by a large margin. \n\n\n2.Appparently, the local CV/Public LB score balance remains a nightmare especially for time series data. A [kaggler](https://www.kaggle.com/competitions/g-research-crypto-forecasting/discussion/323098) discussed at length how he made a validation strategy that resonated well with the private leaderboard. He avoided probing the test data and focused instead on developing a CV scheme that he was confident about. He made use of overlapping folds to extend the data available for training at each fold, to give the model enough data at training time. This overlap was done meticulosuly to avoid target leaks. He finished second.\n\n\n3.I don't know how long this has been known, but for time series data it seems safer to predict change magnitudes, that is (ratio(target at t+1/target at t), especially if a decision tree based model is used. Many solutions used this approach in the [Godaddy](https://www.kaggle.com/competitions/godaddy-microbusiness-density-forecasting) competition. Tree based models suffer from extrapolation incapabality when working with time series, I beleive learning relative changes reduces noise and alleviates that extrapolation problem.\n\n4.Time-aware attention mechanism: Someone taught about using a time aware attention mechanism for a transformer architecture to give more weight to more recent observations in a learning behaviour characterization competition where the goal is to predict the perrfomance of a student over a set of questions in the future based on previously observed data. I haven't been able to recover the link to that discussion but it's noteworthy enough","metadata":{}},{"cell_type":"markdown","source":"#### TABPFN IS HERE TO STAY\n\nI previously alluded to what i think is the paper with the most impact on the community. [TabPFN](https://arxiv.org/abs/2207.01848) performs classification on small datasets in less than a second. The announcement [tweet](https://twitter.com/FrankRHutter/status/1583410856288976898) garnered over 5k likes so i'm guessing that helped in its eventual adoption. TabPFN is based on a [prior fitted network](https://arxiv.org/abs/2112.10510) that has learned to do bayesian inference over a large number of synthetic datasets. The model learns a probability distribution over the data such that it can estimate with good standard how likely it is that a sample belongs to some class. The ICR age related competitions has almost every public kernel using this model in one way or the other. ","metadata":{}},{"cell_type":"markdown","source":"### TOWARDS EFFICIENCY\n\nMany kagglers have good feature engineering and modelling ideas but are limited by memory problems or runtime limits. Here are some tips i learned:\n\n1.[Simulate the Kaggle time series API to alleviate efficiencyproblems](https://www.kaggle.com/code/jagofc/local-api-emulator)\n\n2.[Use FSI for boosted trees](https://developer.nvidia.com/blog/sparse-forests-with-fil/)\n\n3.Cpython is great\n\n4.Numba rocks\n\n5.Use later parts of a sequence for query and key computation for sequence modelling tasks\n\n6.Parquet works","metadata":{}},{"cell_type":"markdown","source":"### CONCLUSION\n\nMany of the papers discusses clearly didn't get the attention of Kagglers but I think there is something worth learning in these papers especially when the feature names are anonymized or we are limited in the assumptions we can make about certain characteristics of the data. ","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}