{ "id": 370046, "title": "9th Place Efficiency Solution", "content": "

Overview

\n

\"\"

\n

For the efficiency track , I trained model which is using pre-trained weights from microsoft/deberta-v3-small.
\nThis model uses combined features generated from mean_pooling + attention_pooling ( using last 3 layers ) + last_layer_cls_token

\n

Model trained on full dataset + pseudo labels generated from FB1 dataset , also created new dataset by applying data augmentation technique like back_translation, word embedding augmentation ( glove) and synonyms augmentation ( wordnet ) on the current competition training dataset.

\n

Final model weights are saved by averaging last 5 checkpoints ( applying the learning from SWA ).

\n

But before making the decision on applying SWA on full training dataset, I trained the same model on 4 folds generated using MultilabelStratifiedKFold and calculated the oof score: 0.4504400497619689

\n

This exercise was helpful for me to keep track on the model architecture performance and selection is based on the best oof score.

\n

Pseudo Labels:

\n

For pseudo labels generation I used the ensemble of four models, which is trained on the using following configuration:

\n\n

\"\"

\n

This ensemble I called as 'ens1_magic' which is giving the oof_score = 0.44848052478266

\n

\"\"

\n

Speedy Inference:

\n

Runtime is also the another most crucial thing to improve along with mcrmse score.

\n

First exported trained model to ONNX .

\n
 def create_onnx_model(input_model, onnx_model_name):\n\n    export_model_path = os.path.join(output_dir, onnx_model_name)\n\n\n    # Enable overwrite to export onnx model and download latest script each time when running this notebook.\n    enable_overwrite = True\n\n\n    # Set model to inference mode, which is required before exporting the model because some operators behave differently in \n    # inference and training mode.\n    input_model.eval()\n    input_model.to(DEVICE)\n\n    if enable_overwrite or not os.path.exists(export_model_path):\n        with torch.no_grad():\n            symbolic_names = {0: \"batch_size\", 1: \"seq_length\"}\n\n            torch.onnx.export(input_model,                                      # model being run\n                              args=tuple(inputs.values()),                      # model input (or a tuple for multiple inputs)\n                              f=export_model_path,                              # where to save the model (can be a file or file-like object)\n                              opset_version=13,                                 # the ONNX version to export the model to\n                              do_constant_folding=True,                         # whether to execute constant folding for optimization\n                              input_names=['input_ids',                         # the model's input names\n                                           'attention_mask', \n                                           ],\n                              output_names=[\"output\"],                    # the model's output names\n                              dynamic_axes={\"input_ids\": symbolic_names,        # variable length axes\n                                            \"attention_mask\" : symbolic_names,\n                                            }\n                             )\n            print(\"Model exported at \", export_model_path)\n\n\ncust_config = AutoConfig.from_pretrained(DOWNLOADED_MODEL_PATH)\ncust_model = Deberta_FBP_MPAPCLS(DOWNLOADED_MODEL_PATH, cust_config)\ncheckpoint = torch.load(current_model['model_base_path'], map_location=torch.device('cpu'))\ncust_model.load_state_dict(checkpoint)\n\n\ncreate_onnx_model(cust_model, 'model.onnx')\n
\n

Second, pre-tokenized all text data using multi-processing before making predictions.

\n
def encode_deberta(text):\n    sample = dict()\n    encodings = TOKENIZER_CFG.tokenizer(\n        text,\n#         return_tensors=\"pt\",\n#         padding=\"max_length\",\n        truncation=True,\n        max_length=TOKENIZER_CFG.max_len,\n    )\n    sample[\"input_ids\"] = np.array(encodings[\"input_ids\"]).astype(np.int32).reshape(1,-1)\n    sample[\"attention_mask\"] = np.array(encodings[\"attention_mask\"]).astype(np.int32).reshape(1,-1)\n    return sample\n\ntest = pd.read_csv('/kaggle/input/feedback-prize-english-language-learning/test.csv')\npool_obj = mp.Pool(N_CORES)\nencoded_texts = pool_obj.map(encode_deberta, test.full_text)\n\nencoded_dataset = FBP_Dataset_Inference_Fast(encoded_texts)\n
\n

Finally, for prediction using multi-processing

\n
model_output = []\nsess_options = onnxruntime.SessionOptions()\n\nsession = onnxruntime.InferenceSession(export_model_path_dv3xs_f3 , sess_options, providers=['CPUExecutionProvider'])\n\ndef speedy_inference(text_idx):\n    data = encoded_dataset[text_idx]\n    ort_inputs = {\n        'input_ids':  data['input_ids'],\n        'attention_mask': data['attention_mask']\n    }\n    ort_outputs = session.run(None, ort_inputs)\n    return ort_outputs[0][0].reshape(-1)\n\n# start = time.time()\n\ntest_idx = list(range(len(test)))\n# test_idx\n\npool_obj = mp.Pool(N_CORES)\n\nmodel_output = pool_obj.map(speedy_inference, test_idx)\ntest_preds = np.array(model_output)\n
\n

all this work complemented each other for faster inference. My model has the runtime of 951 seconds.

\n

What Worked

\n\n

What Didn’t Work

\n

I don't want to say did not work but putting like I am not able to make it work, makes more sense.

\n\n

Important Citations

\n\n

Thanks and Acknowledgements:

\n

I am grateful to the kaggle and competition organizers for hosting this amazing competition.

\n

This is my first efficiency track completion and I have learnt hell lot of things during this competition. So THANK YOU for this new track and looking forward to more and more NLP competition which includes efficiency track.

\n

I also want to shout out to all of the kagglers who brought up amazing, valuable discussion which really helped me to learn new things.

", "comments": [], "comment_count": 0, "messages": [ { "id": 2052882, "postDate": "2022-12-02T15:53:14.520Z", "content": "

Overview

\n

\"\"

\n

For the efficiency track , I trained model which is using pre-trained weights from microsoft/deberta-v3-small.
\nThis model uses combined features generated from mean_pooling + attention_pooling ( using last 3 layers ) + last_layer_cls_token

\n

Model trained on full dataset + pseudo labels generated from FB1 dataset , also created new dataset by applying data augmentation technique like back_translation, word embedding augmentation ( glove) and synonyms augmentation ( wordnet ) on the current competition training dataset.

\n

Final model weights are saved by averaging last 5 checkpoints ( applying the learning from SWA ).

\n

But before making the decision on applying SWA on full training dataset, I trained the same model on 4 folds generated using MultilabelStratifiedKFold and calculated the oof score: 0.4504400497619689

\n

This exercise was helpful for me to keep track on the model architecture performance and selection is based on the best oof score.

\n

Pseudo Labels:

\n

For pseudo labels generation I used the ensemble of four models, which is trained on the using following configuration:

\n\n

\"\"

\n

This ensemble I called as 'ens1_magic' which is giving the oof_score = 0.44848052478266

\n

\"\"

\n

Speedy Inference:

\n

Runtime is also the another most crucial thing to improve along with mcrmse score.

\n

First exported trained model to ONNX .

\n
 def create_onnx_model(input_model, onnx_model_name):\n\n    export_model_path = os.path.join(output_dir, onnx_model_name)\n\n\n    # Enable overwrite to export onnx model and download latest script each time when running this notebook.\n    enable_overwrite = True\n\n\n    # Set model to inference mode, which is required before exporting the model because some operators behave differently in \n    # inference and training mode.\n    input_model.eval()\n    input_model.to(DEVICE)\n\n    if enable_overwrite or not os.path.exists(export_model_path):\n        with torch.no_grad():\n            symbolic_names = {0: \"batch_size\", 1: \"seq_length\"}\n\n            torch.onnx.export(input_model,                                      # model being run\n                              args=tuple(inputs.values()),                      # model input (or a tuple for multiple inputs)\n                              f=export_model_path,                              # where to save the model (can be a file or file-like object)\n                              opset_version=13,                                 # the ONNX version to export the model to\n                              do_constant_folding=True,                         # whether to execute constant folding for optimization\n                              input_names=['input_ids',                         # the model's input names\n                                           'attention_mask', \n                                           ],\n                              output_names=[\"output\"],                    # the model's output names\n                              dynamic_axes={\"input_ids\": symbolic_names,        # variable length axes\n                                            \"attention_mask\" : symbolic_names,\n                                            }\n                             )\n            print(\"Model exported at \", export_model_path)\n\n\ncust_config = AutoConfig.from_pretrained(DOWNLOADED_MODEL_PATH)\ncust_model = Deberta_FBP_MPAPCLS(DOWNLOADED_MODEL_PATH, cust_config)\ncheckpoint = torch.load(current_model['model_base_path'], map_location=torch.device('cpu'))\ncust_model.load_state_dict(checkpoint)\n\n\ncreate_onnx_model(cust_model, 'model.onnx')\n
\n

Second, pre-tokenized all text data using multi-processing before making predictions.

\n
def encode_deberta(text):\n    sample = dict()\n    encodings = TOKENIZER_CFG.tokenizer(\n        text,\n#         return_tensors=\"pt\",\n#         padding=\"max_length\",\n        truncation=True,\n        max_length=TOKENIZER_CFG.max_len,\n    )\n    sample[\"input_ids\"] = np.array(encodings[\"input_ids\"]).astype(np.int32).reshape(1,-1)\n    sample[\"attention_mask\"] = np.array(encodings[\"attention_mask\"]).astype(np.int32).reshape(1,-1)\n    return sample\n\ntest = pd.read_csv('/kaggle/input/feedback-prize-english-language-learning/test.csv')\npool_obj = mp.Pool(N_CORES)\nencoded_texts = pool_obj.map(encode_deberta, test.full_text)\n\nencoded_dataset = FBP_Dataset_Inference_Fast(encoded_texts)\n
\n

Finally, for prediction using multi-processing

\n
model_output = []\nsess_options = onnxruntime.SessionOptions()\n\nsession = onnxruntime.InferenceSession(export_model_path_dv3xs_f3 , sess_options, providers=['CPUExecutionProvider'])\n\ndef speedy_inference(text_idx):\n    data = encoded_dataset[text_idx]\n    ort_inputs = {\n        'input_ids':  data['input_ids'],\n        'attention_mask': data['attention_mask']\n    }\n    ort_outputs = session.run(None, ort_inputs)\n    return ort_outputs[0][0].reshape(-1)\n\n# start = time.time()\n\ntest_idx = list(range(len(test)))\n# test_idx\n\npool_obj = mp.Pool(N_CORES)\n\nmodel_output = pool_obj.map(speedy_inference, test_idx)\ntest_preds = np.array(model_output)\n
\n

all this work complemented each other for faster inference. My model has the runtime of 951 seconds.

\n

What Worked

\n\n

What Didn’t Work

\n

I don't want to say did not work but putting like I am not able to make it work, makes more sense.

\n\n

Important Citations

\n\n

Thanks and Acknowledgements:

\n

I am grateful to the kaggle and competition organizers for hosting this amazing competition.

\n

This is my first efficiency track completion and I have learnt hell lot of things during this competition. So THANK YOU for this new track and looking forward to more and more NLP competition which includes efficiency track.

\n

I also want to shout out to all of the kagglers who brought up amazing, valuable discussion which really helped me to learn new things.

", "rawMarkdown": "# Overview\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1940189%2F3f67af9eb22b46743bd978e1e34ebf34%2FScreenshot%202022-12-02%20at%207.54.24%20PM.png?generation=1669993007713917&alt=media)\n\nFor the efficiency track , I trained model which is using pre-trained weights from `microsoft/deberta-v3-small `. \nThis model uses combined features generated from **mean_pooling + attention_pooling ( using last 3 layers ) + last_layer_cls_token**\n\nModel trained on full dataset + pseudo labels generated from FB1 dataset , also created new dataset by applying data augmentation technique like back_translation, word embedding augmentation ( glove) and synonyms augmentation ( wordnet ) on the current competition training dataset.\n\nFinal model weights are saved by averaging last 5 checkpoints ( applying the learning from SWA ).\n\nBut before making the decision on applying SWA on full training dataset, I trained the same model on 4 folds generated using `MultilabelStratifiedKFold` and calculated the oof score: 0.4504400497619689 \n\nThis exercise was helpful for me to keep track on the model architecture performance and selection is based on the best oof score.\n\n\n# Pseudo Labels:\n\nFor pseudo labels generation I used the ensemble of four models, which is trained on the using following configuration:\n- seed = 42\n- epochs = 4\n- batch_size = 8\n- weight_decay = 0.01\n- warmup_ratio = 0.01\n- n_folds = 5\n- learning_rate = 2e-5\n- max_len = 512\n- max_grad_norm = 1\n- gradient_accumulation_steps = 1\n- layerwise_learning_rate_decay = 0.8\n- classifier_lr = 1e-4\n- reinit_n_layers = 1\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1940189%2Fb7f354c471e5099251592bc1c560aede%2FScreenshot%202022-12-02%20at%208.13.54%20PM.png?generation=1669993096314328&alt=media)\n\nThis ensemble I called as **'ens1_magic'** which is giving the **oof_score = 0.44848052478266**\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1940189%2Fadfd7493af24e16e7faa86eb552ab8b6%2FScreenshot%202022-12-02%20at%208.21.04%20PM.png?generation=1669993069970852&alt=media)\n\n\n# Speedy Inference:\nRuntime is also the another most crucial thing to improve along with mcrmse score. \n\n**First exported trained model to ONNX .**\n\n```\n def create_onnx_model(input_model, onnx_model_name):\n\n export_model_path = os.path.join(output_dir, onnx_model_name)\n\n\n # Enable overwrite to export onnx model and download latest script each time when running this notebook.\n enable_overwrite = True\n\n\n # Set model to inference mode, which is required before exporting the model because some operators behave differently in \n # inference and training mode.\n input_model.eval()\n input_model.to(DEVICE)\n\n if enable_overwrite or not os.path.exists(export_model_path):\n with torch.no_grad():\n symbolic_names = {0: \"batch_size\", 1: \"seq_length\"}\n\n torch.onnx.export(input_model, # model being run\n args=tuple(inputs.values()), # model input (or a tuple for multiple inputs)\n f=export_model_path, # where to save the model (can be a file or file-like object)\n opset_version=13, # the ONNX version to export the model to\n do_constant_folding=True, # whether to execute constant folding for optimization\n input_names=['input_ids', # the model's input names\n 'attention_mask', \n ],\n output_names=[\"output\"], # the model's output names\n dynamic_axes={\"input_ids\": symbolic_names, # variable length axes\n \"attention_mask\" : symbolic_names,\n }\n )\n print(\"Model exported at \", export_model_path)\n\n\ncust_config = AutoConfig.from_pretrained(DOWNLOADED_MODEL_PATH)\ncust_model = Deberta_FBP_MPAPCLS(DOWNLOADED_MODEL_PATH, cust_config)\ncheckpoint = torch.load(current_model['model_base_path'], map_location=torch.device('cpu'))\ncust_model.load_state_dict(checkpoint)\n\n\ncreate_onnx_model(cust_model, 'model.onnx')\n\n```\n\n**Second, pre-tokenized all text data using multi-processing before making predictions.**\n```\ndef encode_deberta(text):\n sample = dict()\n encodings = TOKENIZER_CFG.tokenizer(\n text,\n# return_tensors=\"pt\",\n# padding=\"max_length\",\n truncation=True,\n max_length=TOKENIZER_CFG.max_len,\n )\n sample[\"input_ids\"] = np.array(encodings[\"input_ids\"]).astype(np.int32).reshape(1,-1)\n sample[\"attention_mask\"] = np.array(encodings[\"attention_mask\"]).astype(np.int32).reshape(1,-1)\n return sample\n\ntest = pd.read_csv('/kaggle/input/feedback-prize-english-language-learning/test.csv')\npool_obj = mp.Pool(N_CORES)\nencoded_texts = pool_obj.map(encode_deberta, test.full_text)\n\nencoded_dataset = FBP_Dataset_Inference_Fast(encoded_texts)\n```\n\n**Finally, for prediction using multi-processing**\n```\nmodel_output = []\nsess_options = onnxruntime.SessionOptions()\n\nsession = onnxruntime.InferenceSession(export_model_path_dv3xs_f3 , sess_options, providers=['CPUExecutionProvider'])\n\ndef speedy_inference(text_idx):\n data = encoded_dataset[text_idx]\n ort_inputs = {\n 'input_ids': data['input_ids'],\n 'attention_mask': data['attention_mask']\n }\n ort_outputs = session.run(None, ort_inputs)\n return ort_outputs[0][0].reshape(-1)\n\n# start = time.time()\n\ntest_idx = list(range(len(test)))\n# test_idx\n\npool_obj = mp.Pool(N_CORES)\n\nmodel_output = pool_obj.map(speedy_inference, test_idx)\ntest_preds = np.array(model_output)\n```\n\nall this work complemented each other for faster inference. **My model has the runtime of 951 seconds.**\n\n\n# What Worked\n- Pseudo Labelling\n- Data Augmentation\n- Training the model on complete dataset\n- SWA ( average of last 5 checkpoints )\n\n# What Didn’t Work\nI don't want to say did not work but putting like I am not able to make it work, makes more sense.\n- AWP\n- FGM\n- Quantization\n\n# Important Citations\n- Data Augmentation: [https://github.com/makcedward/nlpaug/blob/master/example/textual_augmenter.ipynb](https://github.com/makcedward/nlpaug/blob/master/example/textual_augmenter.ipynb)\n- [How To Build an Efficient NLP Model (Article)](https://wandb.ai/darek/fbck/reports/How-To-Build-an-Efficient-NLP-Model--VmlldzoyNTE5MDEx?accessToken=pmm41mpdkxif0lsbm927tfxj947to4gbd0nrgjjq9rdoq1c4jr3kruf993ys5kpg)\n\n# Thanks and Acknowledgements:\nI am grateful to the kaggle and competition organizers for hosting this amazing competition. \n\nThis is my first efficiency track completion and I have learnt hell lot of things during this competition. So THANK YOU for this new track and looking forward to more and more NLP competition which includes efficiency track. \n\nI also want to shout out to all of the kagglers who brought up amazing, valuable discussion which really helped me to learn new things. \n", "votes": 12 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:04:44Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9157547, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/shobhit-upadhyaya-9th-place-efficiency-solution" }