# 9th Place Efficiency Solution Overview ![](assets/836d66038c789e2a6af980a1.png) For the efficiency track , I trained model which is using pre-trained weights from microsoft/deberta-v3-small . This model uses combined features generated from mean_pooling + attention_pooling ( using last 3 layers ) + last_layer_cls_token Model trained on full dataset + pseudo labels generated from FB1 dataset , also created new dataset by applying data augmentation technique like back_translation, word embedding augmentation ( glove) and synonyms augmentation ( wordnet ) on the current competition training dataset. Final model weights are saved by averaging last 5 checkpoints ( applying the learning from SWA ). But before making the decision on applying SWA on full training dataset, I trained the same model on 4 folds generated using MultilabelStratifiedKFold and calculated the oof score: 0.4504400497619689 This exercise was helpful for me to keep track on the model architecture performance and selection is based on the best oof score. Pseudo Labels: For pseudo labels generation I used the ensemble of four models, which is trained on the using following configuration: seed = 42 epochs = 4 batch_size = 8 weight_decay = 0.01 warmup_ratio = 0.01 n_folds = 5 learning_rate = 2e-5 max_len = 512 max_grad_norm = 1 gradient_accumulation_steps = 1 layerwise_learning_rate_decay = 0.8 classifier_lr = 1e-4 reinit_n_layers = 1 ![](assets/5ec7ad2a2eabfe1723777e75.png) This ensemble I called as 'ens1_magic' which is giving the oof_score = 0.44848052478266 ![](assets/ae0bcb1951ee2f2644d2226c.png) Speedy Inference: Runtime is also the another most crucial thing to improve along with mcrmse score. First exported trained model to ONNX . def create_onnx_model(input_model, onnx_model_name): export_model_path = os.path.join(output_dir, onnx_model_name) # Enable overwrite to export onnx model and download latest script each time when running this notebook. enable_overwrite = True # Set model to inference mode, which is required before exporting the model because some operators behave differently in # inference and training mode. input_model.eval() input_model.to(DEVICE) if enable_overwrite or not os.path.exists(export_model_path): with torch.no_grad(): symbolic_names = {0: "batch_size", 1: "seq_length"} torch.onnx.export(input_model, # model being run args=tuple(inputs.values()), # model input (or a tuple for multiple inputs) f=export_model_path, # where to save the model (can be a file or file-like object) opset_version=13, # the ONNX version to export the model to do_constant_folding=True, # whether to execute constant folding for optimization input_names=['input_ids', # the model's input names 'attention_mask', ], output_names=["output"], # the model's output names dynamic_axes={"input_ids": symbolic_names, # variable length axes "attention_mask" : symbolic_names, } ) print("Model exported at ", export_model_path) cust_config = AutoConfig.from_pretrained(DOWNLOADED_MODEL_PATH) cust_model = Deberta_FBP_MPAPCLS(DOWNLOADED_MODEL_PATH, cust_config) checkpoint = torch.load(current_model['model_base_path'], map_location=torch.device('cpu')) cust_model.load_state_dict(checkpoint) create_onnx_model(cust_model, 'model.onnx') Second, pre-tokenized all text data using multi-processing before making predictions. def encode_deberta(text): sample = dict() encodings = TOKENIZER_CFG.tokenizer( text, # return_tensors="pt", # padding="max_length", truncation=True, max_length=TOKENIZER_CFG.max_len, ) sample["input_ids"] = np.array(encodings["input_ids"]).astype(np.int32).reshape(1,-1) sample["attention_mask"] = np.array(encodings["attention_mask"]).astype(np.int32).reshape(1,-1) return sample test = pd.read_csv('/kaggle/input/feedback-prize-english-language-learning/test.csv') pool_obj = mp.Pool(N_CORES) encoded_texts = pool_obj.map(encode_deberta, test.full_text) encoded_dataset = FBP_Dataset_Inference_Fast(encoded_texts) Finally, for prediction using multi-processing model_output = [] sess_options = onnxruntime.SessionOptions() session = onnxruntime.InferenceSession(export_model_path_dv3xs_f3 , sess_options, providers=['CPUExecutionProvider']) def speedy_inference(text_idx): data = encoded_dataset[text_idx] ort_inputs = { 'input_ids': data['input_ids'], 'attention_mask': data['attention_mask'] } ort_outputs = session.run(None, ort_inputs) return ort_outputs[0][0].reshape(-1) # start = time.time() test_idx = list(range(len(test))) # test_idx pool_obj = mp.Pool(N_CORES) model_output = pool_obj.map(speedy_inference, test_idx) test_preds = np.array(model_output) all this work complemented each other for faster inference. My model has the runtime of 951 seconds. What Worked Pseudo Labelling Data Augmentation Training the model on complete dataset SWA ( average of last 5 checkpoints ) What Didn’t Work I don't want to say did not work but putting like I am not able to make it work, makes more sense. AWP FGM Quantization Important Citations Data Augmentation: [https://github.com/makcedward/nlpaug/blob/master/example/textual_augmenter.ipynb](https://github.com/makcedward/nlpaug/blob/master/example/textual_augmenter.ipynb) [How To Build an Efficient NLP Model (Article)](https://wandb.ai/darek/fbck/reports/How-To-Build-an-Efficient-NLP-Model--VmlldzoyNTE5MDEx?accessToken=pmm41mpdkxif0lsbm927tfxj947to4gbd0nrgjjq9rdoq1c4jr3kruf993ys5kpg) Thanks and Acknowledgements: I am grateful to the kaggle and competition organizers for hosting this amazing competition. This is my first efficiency track completion and I have learnt hell lot of things during this competition. So THANK YOU for this new track and looking forward to more and more NLP competition which includes efficiency track. I also want to shout out to all of the kagglers who brought up amazing, valuable discussion which really helped me to learn new things.