# Solution writeup, Efficiency leaderboard #24. My first kaggle competition, top 50!

24th place solution, efficiency leaderboard!
This was my first kaggle competition and I was aiming for the efficiency prize. I think I did well to be in the top 50!
You can see my solution's train code
[here](https://www.kaggle.com/code/wisawesome/24th-place-eff-prize-feedback-prize-ell-train)
and the infer code
[here](https://www.kaggle.com/wisawesome/24th-place-eff-prize-feedback-prize-ell-infer)
. Note that I forgot to set the seed for these, hence you may obtain slightly different results to the ones I obtained.
The solution
I used iterative stratification to get 5 folds from the training data. I then trained catboost and svr models for each of the 6 targets (cohesion,syntax,vocabulary,phraseology,grammar,conventions) independently using the embeddings (mean pooling, max pooling and all 12 hidden state embeddings) extracted from DeBERTa V3 xsmall.
I used published code from the following notebooks to extract the embeddings and to generate the folds using iterative stratification:
[https://www.kaggle.com/code/lextoumbourou/feedback3-eda-hf-custom-trainer-sift#Mean,-Min-and-Max-Pooling-Implementations](url)
[https://www.kaggle.com/code/tafreen/notebook-ell](url)
[https://www.kaggle.com/code/brandonhu0215/feedback-deberta-large-lb0-619](url)
[https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x](url)
[https://www.kaggle.com/code/kvsnoufal/lb0-46-gb-debertaembedding](url)
Some functions used to extract features from DeBERTa V3 xsmall:
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output.last_hidden_state.detach().cpu()
    input_mask_expanded = (
        attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    )
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(
        input_mask_expanded.sum(1), min=1e-9
    )
def MaxPooling(output, attention_mask):  
    last_hidden_state = output[0]
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(last_hidden_state.size()).float()
    last_hidden_state[input_mask_expanded == 0] = -1e9  # Set padding tokens to large negative value
    max_embeddings = torch.max(last_hidden_state, 1)[0]
    return max_embeddings
def get_embeddings(MODEL_NM='', MAX=640, BATCH_SIZE=4, verbose=True, use_cuda=use_cuda):
    global tokenizer, MAX_LEN
    if use_cuda:
        DEVICE="cuda"
    else:
        DEVICE="cpu"
    config = AutoConfig.from_pretrained(MODEL_NM)
    config.update({'output_hidden_states':True})
    model = AutoModel.from_pretrained( MODEL_NM, config=config )
    tokenizer = AutoTokenizer.from_pretrained( MODEL_NM )
    MAX_LEN = MAX
    model = model.to(DEVICE)
    model.eval()
    all_train_text_feats_esr = []
    all_train_text_feats_mean = []
    for batch in tqdm(embed_dataloader_tr,total=len(embed_dataloader_tr)):
        input_ids = batch["input_ids"].to(DEVICE)
        attention_mask = batch["attention_mask"].to(DEVICE)
        with torch.no_grad():
            model_output = model(input_ids=input_ids,attention_mask=attention_mask)
        #sentence_embeddings = mean_pooling(model_output, attention_mask.detach().cpu())
        ## Normalize the embeddings
        #sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)
        #sentence_embeddings =  sentence_embeddings.squeeze(0).detach().cpu().numpy()
        mean_pool_embeddings = mean_pooling(model_output, attention_mask.detach().cpu())
        ## Normalize the embeddings
        mean_pool_embeddings = F.normalize(mean_pool_embeddings, p=2, dim=1)
        mean_pool_embeddings =  mean_pool_embeddings.squeeze(0).detach().cpu().numpy()
        max_pool_embeddings = MaxPooling(model_output, attention_mask.detach().cpu())
        ## Normalize the embeddings
        max_pool_embeddings = F.normalize(max_pool_embeddings, p=2, dim=1)
        max_pool_embeddings =  max_pool_embeddings.squeeze(0).detach().cpu().numpy()
        #cls_pool_embeddings = CLS_embeddings(model_output)
        ## Normalize the embeddings
        #cls_pool_embeddings = F.normalize(cls_pool_embeddings, p=2, dim=1)
        #cls_pool_embeddings =  cls_pool_embeddings.squeeze(0).detach().cpu().numpy()
        # Concatenate last 4 hidden layers
        all_hidden_states = torch.stack(model_output.hidden_states)
        concatenate_pooling = torch.cat(
            (all_hidden_states[-1], all_hidden_states[-2], all_hidden_states[-3], all_hidden_states[-4], all_hidden_states[-5], all_hidden_states[-6], all_hidden_states[-7], all_hidden_states[-8], all_hidden_states[-9], all_hidden_states[-10], all_hidden_states[-11], all_hidden_states[-12]),-1
        )
        concatenate_pooling = concatenate_pooling[:, 0]
        concatenate_pooling = F.normalize(concatenate_pooling, p=2, dim=1)
        concatenate_pooling = concatenate_pooling.squeeze(0).detach().cpu().numpy()
        #embeddings_concat = np.concatenate([mean_pool_embeddings, max_pool_embeddings, cls_pool_embeddings, concatenate_pooling], axis=1)
        #embeddings_concat = np.concatenate([mean_pool_embeddings, concatenate_pooling], axis=1)
        embeddings_concat = np.concatenate([mean_pool_embeddings, max_pool_embeddings, concatenate_pooling], axis=1)
        #all_train_text_feats.extend(sentence_embeddings)
        all_train_text_feats_esr.extend(embeddings_concat)
        all_train_text_feats_mean.extend(mean_pool_embeddings)
    all_train_text_feats_esr = np.array(all_train_text_feats_esr)
    all_train_text_feats_mean = np.array(all_train_text_feats_mean)
    #if verbose:
    #    print('Train embeddings shape',all_train_text_feats.shape)
    te_text_feats_esr = []
    te_text_feats_mean = []
    for batch in tqdm(embed_dataloader_te,total=len(embed_dataloader_te)):
        input_ids = batch["input_ids"].to(DEVICE)
        attention_mask = batch["attention_mask"].to(DEVICE)
        with torch.no_grad():
            model_output = model(input_ids=input_ids,attention_mask=attention_mask)
        #sentence_embeddings = mean_pooling(model_output, attention_mask.detach().cpu())
        ## Normalize the embeddings
        #sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)
        #sentence_embeddings =  sentence_embeddings.squeeze(0).detach().cpu().numpy()
        mean_pool_embeddings = mean_pooling(model_output, attention_mask.detach().cpu())
        ## Normalize the embeddings
        mean_pool_embeddings = F.normalize(mean_pool_embeddings, p=2, dim=1)
        mean_pool_embeddings =  mean_pool_embeddings.squeeze(0).detach().cpu().numpy()
        max_pool_embeddings = MaxPooling(model_output, attention_mask.detach().cpu())
        ## Normalize the embeddings
        max_pool_embeddings = F.normalize(max_pool_embeddings, p=2, dim=1)
        max_pool_embeddings =  max_pool_embeddings.squeeze(0).detach().cpu().numpy()
        #cls_pool_embeddings = CLS_embeddings(model_output)
        ## Normalize the embeddings
        #cls_pool_embeddings = F.normalize(cls_pool_embeddings, p=2, dim=1)
        #cls_pool_embeddings =  cls_pool_embeddings.squeeze(0).detach().cpu().numpy()
        # Concatenate last 4 hidden layers
        all_hidden_states = torch.stack(model_output.hidden_states)
        concatenate_pooling = torch.cat(
            (all_hidden_states[-1], all_hidden_states[-2], all_hidden_states[-3], all_hidden_states[-4], all_hidden_states[-5], all_hidden_states[-6], all_hidden_states[-7], all_hidden_states[-8], all_hidden_states[-9], all_hidden_states[-10], all_hidden_states[-11], all_hidden_states[-12]),-1
        )
        concatenate_pooling = concatenate_pooling[:, 0]
        concatenate_pooling = F.normalize(concatenate_pooling, p=2, dim=1)
        concatenate_pooling = concatenate_pooling.squeeze(0).detach().cpu().numpy()
        #embeddings_concat = np.concatenate([mean_pool_embeddings, max_pool_embeddings, cls_pool_embeddings, concatenate_pooling], axis=1)
        #embeddings_concat = np.concatenate([mean_pool_embeddings, concatenate_pooling], axis=1)
        embeddings_concat = np.concatenate([mean_pool_embeddings, max_pool_embeddings, concatenate_pooling], axis=1)
        #te_text_feats.extend(sentence_embeddings)
        te_text_feats_esr.extend(embeddings_concat)
        te_text_feats_mean.extend(mean_pool_embeddings)
    te_text_feats_esr = np.array(te_text_feats_esr)
    te_text_feats_mean = np.array(te_text_feats_mean)
    #if verbose:
    #    print('Test embeddings shape',te_text_feats.shape)
    return all_train_text_feats_esr, all_train_text_feats_mean, te_text_feats_esr, te_text_feats_mean
Targets cohesion, syntax, vocabulary, phraseology, and conventions were then targeted using catboost or SVR models using the features in te_text_feats_esr (which contains mean pooling, max pooling, and all 12 hidden states). Grammar was targeted using an SVR model using the features in te_text_feats_mean (which just contains the mean pooling).
The catboost and SVR models had their hyperparameters optimized using optuna. During training, the training set was split into 5 folds using iterative stratification. The best performing hyperparameters (as measured by mean performance on all 5 folds) were then used to train one model (either a catboost or SVR model, depending on which performed better on the cv) for each target using all available training data.
What Worked
[High Impact] Idea
I quickly realized that the efficiency metric was designed well, meaning that neither the leaderboard score component nor the computation time penalty dominate. In other words, in order to do well in the efficiency competition the submission has to have a competitive score, but must also have a low computation time.
I settled on using DeBERTa V3 xsmall from huggingface (
[https://huggingface.co/microsoft/deberta-v3-xsmall](url)
) which provided a good score at a low computation cost. All other transformers either took too long or did not improve my cross-validation score.
I set max_len to 400. I tried various max_len options from 100 to 640, timed them, and computed the cross-validation score. As expected, higher max_len resulted in better scores, but also longer computation times. It was hard to find an exact tradeoff between computation time and score improvement: it was impossible to tell exactly what the leaderboard scores would be at the end of the competition; full precision was not provided; and there was variation in computation times between runs on the training data. Through experimentation and some back-of-the-napkin calculations I estimated that a 0.01 improvement in the score was roughly equal to 25 minutes of computation time. This tradeoff guided my decision to choose a max_len of 400: longer lengths improved the score, but the time penalty was too high; lower max_len lowered the score but could not be justified on the basis of the computation time improvement.
I did not train any transformer models because I did not even know what a transformer was before this competition. Instead, I focused on optimizing hyperparameters using optuna and selecting the correct max_len for the features extracted from DeBERTa V3 xsmall.
I quickly found that catboost and SVR performed the best in my setup, and focused extensively on optimizing hyperparameters for those two methods.
What Didn’t Work
xgboost and lightgbm. I was surprised by the significantly worse performance of these two methods in comparison to catboost. Catboost's ordered boosting mode and symmetric tree method dominated the above but did take longer to train.
I could not get ensembles to improve my score, but on reflection I should have spent more time trying to do so. I believe an ensemble of catboost and svr models would have dominated my score, but on the public leaderboard that was not the case, so the idea was prematurely dropped.
Inclusion of embeddings from any other transformer model. I was really dismayed by this. There are many "tiny" models from which it only takes a minute to extract all encodings, yet the inclusion of these encodings did not improve my score.
Taking a risk by rounding to the nearest possible score did not pay off. For instance, if the model thought grammar was 3.55, rounding to 3.5 did caused a slight decrease in the score. I would have been better off if I did not round to the nearest possible score.
Important Citations:
[https://www.kaggle.com/code/lextoumbourou/feedback3-eda-hf-custom-trainer-sift#Mean,-Min-and-Max-Pooling-Implementations](url)
[https://www.kaggle.com/code/tafreen/notebook-ell](url)
[https://www.kaggle.com/code/brandonhu0215/feedback-deberta-large-lb0-619](url)
[https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x](url)
[https://www.kaggle.com/code/kvsnoufal/lb0-46-gb-debertaembedding](url)
Thanks and Acknowledgements
I would like to thank the following kagglers for publishing their code which helped me reach my solution:
[@lextoumbourou](https://www.kaggle.com/lextoumbourou)
[@tafreen](https://www.kaggle.com/tafreen)
[@brandonhu](https://www.kaggle.com/brandonhu)
[@cdeotte](https://www.kaggle.com/cdeotte)
[@kvsnoufal](https://www.kaggle.com/kvsnoufal)
I would like to thank the organizers as well as Kaggle for hosting this excellent competition.
See you all in the next one!