{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":67223,"databundleVersionId":7463838,"sourceType":"competition"}],"dockerImageVersionId":30627,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install pytorch_tabular --quiet","metadata":{"execution":{"iopub.status.busy":"2024-01-16T13:43:21.012009Z","iopub.execute_input":"2024-01-16T13:43:21.012874Z","iopub.status.idle":"2024-01-16T13:43:43.409772Z","shell.execute_reply.started":"2024-01-16T13:43:21.012840Z","shell.execute_reply":"2024-01-16T13:43:43.408794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://pytorch-tabular.readthedocs.io/en/latest/imgs/pytorch_tabular_logo.png)\n\nPyTorch Tabular is a powerful library that aims to simplify and popularize the application of deep learning techniques to tabular data. Tabular deep learning has gained significant importance in the field of machine learning due to its ability to handle structured data, such as data in spreadsheets or databases. However, working with tabular data can be challenging, requiring expertise in both deep learning and data preprocessing.\n\nThis is where PyTorch Tabular comes in. Built on the shoulders of giants like PyTorch, PyTorch Lightning, and pandas, PyTorch Tabular offers a low resistance usability, making it accessible to both real-world use cases and research projects. The library's core principles revolve around easy customization, allowing users to tailor their models and pipelines to specific requirements. Moreover, PyTorch Tabular provides scalable and efficient tooling, making it easier to deploy models in production environments. The underlying goodness of PyTorch makes designing deep learning architectures pythonic and intuitive, while PyTorch Lightning simplifies the training process. pandas is the de-facto standard for working with tabular data, and PyTorch Tabular leverages its strengths to simplify the preprocessing of tabular data. With PyTorch Tabular, data scientists and researchers can focus on the core aspects of their work, while the library takes care of the underlying complexities, enabling efficient and effective tabular deep learning.\n\n<img src=\"https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/github.svg\" width=\"15\" height=\"15\"> [Github](https://github.com/manujosephv/pytorch_tabular)\n<img src=\"https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/solid/book.svg\" width=\"15\" height=\"15\"> [Documentation](https://pytorch-tabular.readthedocs.io/en/latest/)\n<img src=\"https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/solid/chalkboard-user.svg\" width=\"15\" height=\"15\"> [Tutorials](https://pytorch-tabular.readthedocs.io/en/latest/tutorials/01-Approaching%20Any%20Tabular%20Problem%20with%20PyTorch%20Tabular/)\n\n<div class=\"alert alert-block alert-info\"> <b>NOTE</b> <br></br> I strongly advise you to go through the Tutorials on the documentation page, especially \"Approaching any Tabular Problem using PyTorch Tabular\". </div>","metadata":{}},{"cell_type":"code","source":"import numpy as np, pandas as pd\nimport os\nimport json","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-01-16T13:45:25.809912Z","iopub.execute_input":"2024-01-16T13:45:25.811125Z","iopub.status.idle":"2024-01-16T13:45:26.188905Z","shell.execute_reply.started":"2024-01-16T13:45:25.811074Z","shell.execute_reply":"2024-01-16T13:45:26.188136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Read the Data\n\nData is a set of 4 Tasks:\n\n- Numerical Features + Regression (7 Datasets)\n- Numerical Features + Classification (17 Datasets)\n- Numerical + Categorical Features + Regression (19 Datasets)\n- Numerical + Categorical Features + Classification (16 Datasets)\n\nAnd each task has many datasets.","metadata":{}},{"cell_type":"code","source":"path = '/kaggle/input/bench-tab-v1/'\n\ntasks = json.load(open(f'{path}train/dict_task.json'))\nsuits = json.load(open(f'{path}train/dict_suit.json'))\n\nDEBUG = False\nsuits = {'334': [361110, 361111], '335': [361093]} if DEBUG else suits\nn_seeds = 2 if DEBUG else 3\n\ndef prepare_sub(data, identifier):\n    data = pd.DataFrame(data)\n    data.columns = ['target']\n    data['ID'] =  identifier + '_' + data.index.astype(str)\n    columns = ['ID','target']\n    return data[columns]","metadata":{"execution":{"iopub.status.busy":"2024-01-16T14:09:44.351556Z","iopub.execute_input":"2024-01-16T14:09:44.351893Z","iopub.status.idle":"2024-01-16T14:09:44.359988Z","shell.execute_reply.started":"2024-01-16T14:09:44.351867Z","shell.execute_reply":"2024-01-16T14:09:44.359089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Config","metadata":{}},{"cell_type":"code","source":"# Select which TASK. Can be either a single TASK or a list of TASK\nTASK = \"334\"\nSEED = 42\nBATCH_SIZE = 1024\nMAX_EPOCHS = 100\nEARLY_STOPPING = True\nOPTIMIZER = \"AdamW\"\nWEIGHT_DECAY = 1e-6\nLR_SCHEDULER = \"CosineAnnealingWarmRestarts\"\nLR_SCHEDULER_PARAMS = {\"T_0\": 10, \"T_mult\": 1, \"eta_min\": 1e-5}\nLEARNING_RATE = 1e-3\n\nMODEL_PARAMS = {\"gflu_stages\":10}\n","metadata":{"execution":{"iopub.status.busy":"2024-01-16T14:12:02.917822Z","iopub.execute_input":"2024-01-16T14:12:02.918200Z","iopub.status.idle":"2024-01-16T14:12:02.923992Z","shell.execute_reply.started":"2024-01-16T14:12:02.918170Z","shell.execute_reply":"2024-01-16T14:12:02.923003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Available Models in PyTorch Tabular","metadata":{}},{"cell_type":"code","source":"import pytorch_tabular\npytorch_tabular.available_models()","metadata":{"execution":{"iopub.status.busy":"2024-01-16T14:09:24.565419Z","iopub.execute_input":"2024-01-16T14:09:24.565804Z","iopub.status.idle":"2024-01-16T14:09:24.572229Z","shell.execute_reply.started":"2024-01-16T14:09:24.565776Z","shell.execute_reply":"2024-01-16T14:09:24.571231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Details on these models is available [here](https://pytorch-tabular.readthedocs.io/en/latest/models/)\n\nChoose one of the models from the list> Let's choose GANDALF, for now.","metadata":{}},{"cell_type":"markdown","source":"## Importing the required classes","metadata":{}},{"cell_type":"code","source":"from pytorch_tabular import TabularModel\nfrom pytorch_tabular.models import (\n    GANDALFConfig,\n)\nfrom pytorch_tabular.config import DataConfig, OptimizerConfig, TrainerConfig\nfrom pytorch_tabular.models.common.heads import LinearHeadConfig","metadata":{"execution":{"iopub.status.busy":"2024-01-16T14:09:26.069664Z","iopub.execute_input":"2024-01-16T14:09:26.070039Z","iopub.status.idle":"2024-01-16T14:09:26.075182Z","shell.execute_reply.started":"2024-01-16T14:09:26.070010Z","shell.execute_reply":"2024-01-16T14:09:26.074114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = []\nfor task in TASK if isinstance(TASK, list) else [TASK]:\n    for suit in suits[task]:\n        identifier = f'{task}_{suit}'\n        task_type = tasks[task]\n        var_type = task_type['features']\n        target_type = task_type['target']\n        print(identifier, task_type, var_type, target_type)\n        # Read the data\n        X = pd.read_parquet(f'{path}train/{identifier}/data_train.parquet')\n        X_test = pd.read_parquet(f'{path}train/{identifier}/data_test.parquet')\n        y = pd.read_parquet(f'{path}train/{identifier}/target_train.parquet')\n\n        names = json.load(open(f'{path}train/{identifier}/attribute_names.json'))\n        cat_indicator = json.load(open(f'{path}train/{identifier}/categorical_indicator.json'))\n        # Find categorical and numerical features\n        cat_cols = [n for n, c in zip(names, cat_indicator) if c]\n        num_cols = list(set(names)-set(cat_cols))\n        # Assigning classification or regression as task based on target type\n        task_pt = \"classification\" if target_type!=\"num\" else \"regression\"\n        # PyTorch Tabular expects s single dataframe as input\n        X['target'] = y.values\n        del y\n        # Cat Cols as Categorical dtype messes with the categorical encoding\n        X[cat_cols] = X[cat_cols].astype(str)\n        X_test[cat_cols] = X_test[cat_cols].astype(str)\n        # Define Configs | Check API for other options\n        data_config = DataConfig(\n            target=[\n                \"target\"\n            ],\n            continuous_cols=num_cols,\n            categorical_cols=cat_cols,\n            num_workers=3\n        )\n        trainer_config = TrainerConfig(\n            batch_size=BATCH_SIZE,\n            max_epochs=MAX_EPOCHS,\n            early_stopping=\"valid_loss\" if EARLY_STOPPING else None,  # Monitor valid_loss for early stopping\n            early_stopping_mode=\"min\",  # Set the mode as min because for val_loss, lower is better\n            early_stopping_patience=5,  # No. of epochs of degradation training will wait before terminating\n            checkpoints=\"valid_loss\",  # Save best checkpoint monitoring val_loss\n            load_best=True,  # After training, load the best checkpoint\n            progress_bar=\"none\",  # Turning off Progress bar\n            trainer_kwargs=dict(enable_model_summary=False),  # Turning off model summary\n#             fast_dev_run=True\n        )\n        optimizer_config = OptimizerConfig(\n            optimizer=OPTIMIZER,\n            optimizer_params={\"weight_decay\": WEIGHT_DECAY},\n            lr_scheduler=LR_SCHEDULER,\n            lr_scheduler_params=LR_SCHEDULER_PARAMS,\n        )\n        head_config = LinearHeadConfig(\n            layers=\"\",\n            dropout=0.1,\n            initialization=(  # No additional layer in head, just a mapping layer to output_dim\n                \"kaiming\"\n            ),\n        ).__dict__  # Convert to dict to pass to the model config (OmegaConf doesn't accept objects)\n        model_config = GANDALFConfig(\n            task=task_pt,\n            learning_rate=LEARNING_RATE,\n            head=\"LinearHead\",  # Linear Head\n            head_config=head_config,  # Linear Head Config\n            **MODEL_PARAMS\n        )\n        # Initialize the Tabular Model\n        tabular_model = TabularModel(\n            data_config=data_config,\n            model_config=model_config,\n            optimizer_config=optimizer_config,\n            trainer_config=trainer_config,\n            verbose=True,\n            suppress_lightning_logger=False\n        )\n        # If you have separate VAL defined, you can pass in that. If not, PyTorch Tabular will automatically take a sample out of Training to use as Validation\n        tabular_model.fit(\n            train=X,\n            seed=SEED\n        )\n        # Predict on Test Data\n        prediction = tabular_model.predict(X_test, progress_bar=None)\n        if task_pt ==\"classification\":\n            # Taking probabilities of class 1\n            preds = prediction.iloc[:,1].values\n        else:\n            # Taking absolute prediction\n            preds = prediction.target_prediction.values\n\n\n    preds = pd.DataFrame(preds)\n    preds.index = X_test.index\n\n    # appends empty tests / solutions\n    sub.append(prepare_sub(preds, identifier))\n","metadata":{"execution":{"iopub.status.busy":"2024-01-16T14:12:42.769167Z","iopub.execute_input":"2024-01-16T14:12:42.769534Z","iopub.status.idle":"2024-01-16T14:13:11.984814Z","shell.execute_reply.started":"2024-01-16T14:12:42.769507Z","shell.execute_reply":"2024-01-16T14:13:11.983621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.concat(sub).reset_index(drop=True)\ndisplay(sub.head())\ntag = TASK if isinstance(TASK, str) else \"|\".join(TASK)\nsub.to_csv(f'submission_{tag}.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-01-16T14:14:59.155230Z","iopub.execute_input":"2024-01-16T14:14:59.156082Z","iopub.status.idle":"2024-01-16T14:14:59.177753Z","shell.execute_reply.started":"2024-01-16T14:14:59.156052Z","shell.execute_reply":"2024-01-16T14:14:59.176907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Final Submission\n\nYou will have to concatenate all the submission files over different runs and submit a single file to the competition","metadata":{}}]}