# 2nd place solution (updated with code/notebooks)

First, thanks to all the organizers and kaggle staff and congrats to all the winners and thanks to my amazing teammates
[@kneroma](https://www.kaggle.com/kneroma)
and
[@tikutiku](https://www.kaggle.com/tikutiku)
!  With this gold medal,
[@tikutiku](https://www.kaggle.com/tikutiku)
and I finally become competition GMs. We have released our code/notebooks:
code:
[https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution](https://github.com/Shujun-He/TeamSKT-Feedback-Prize---Predicting-Effective-Arguments-2nd-Place-solution)
2nd logloss notebook:
[https://www.kaggle.com/code/kneroma/kkiller-final-fpzeff-team-work-inference?scriptVersionId=104008824](https://www.kaggle.com/code/kneroma/kkiller-final-fpzeff-team-work-inference?scriptVersionId=104008824)
3rd place efficiency notebook:
[https://www.kaggle.com/code/kneroma/fpzeff-fast-lb-efgp-inference?scriptVersionId=104922605](https://www.kaggle.com/code/kneroma/fpzeff-fast-lb-efgp-inference?scriptVersionId=104922605)
Below is a summary of our solution. Our best private is 0.553 and our best selected private is 0.554. Please feel free to ask if you have any questions.
Transformer modeling
Each of our team members has their own training pipeline for transformer models. On a high level, our transformer models look at the entirety of each essay and output predictions of effectiveness for each discourse either via pooling of discourse tokens or a classification token added to the front of the each discourse. Importantly, directly inputting essays results in a situation where the model is not 100% sure about where it needs to make predictions, so to circumvent this, we use either a prompt (i.e. concat
f'({discourse_type} start)'
and
f'({discourse_type} end)'
to the beginning and end of each discourse to signal where predictions need to be made) or simply concat special tokens added to the tokenizer instead (i.e.
f'<{discourse_type}>'
and
f'<{discourse_type}\>'
). You can find an example below with a highlighted segment.
![example](assets/de2360df131912fd51f17715.png)
Encoders
Deberta worked the best since (IMO) it supports unlimited input length and uses disentangled attention with relative positional embedding; in fact, our ensemble consists entirely of deberta variants. For me, it was also helpful to add a GRU/LSTM after pooling on the pooled discourse representations. Tom used my SlidingWindowTransformerModel (
[https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235)
) from the last Feedback competiion, which stablized training for him.
Pretraining
Kkiller used pretrained weights from his solution in the last competiion (
[https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313478](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313478)
) and Tom and I found pretrained tascj0 models to be good starting points. We used some of the weights that tascj0 released after the last Feedback competition and Tom also pretrained some new ones on his own. Please checkout out tascj0's solution post if you'd like to learn more (
[https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424)](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424))
.
In addition, Tom used MLM for some of his models.  Further, some of our models simply used huggingface weights.
Max sequence length
I used a max sequence length of 1280 in both training and inference, since I found that 99.9% of discourses fall within that range, whereas my teammates used up to around 1800 during inference and as low as 648 in training.
Pseudo labeling
Pseudo labeling is an integral part of all our solution. We use essays from the training set of last Feedback competition that are also not present in the training set of this competition. Our procedure is as follows:
Train model with gt labels
Make predictions for old data (around 11000 essays) with each fold model
Retrain model with crossentropy on pseudo label probabilities (not discretized) generated by previous model trained on the same fold data: 3 epochs on pl labels only first and then 3 more epochs on gt labels only
Repeat from step 2
For my pipeline, I saw improvement until 5 rounds of the above procedure. For Tom, it was only helpful for one round and kkiller did not have enough time to try multi-round pl.
Stacking
Stacking provides significantly improvement in both cv/lb (around 0.004). Our stacking framework is primarily inspired by my team's solution (
[https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235)
) in the previous feedback competition. In addition to predicted probabilities outputted by the transformer models, we also utilized the token probabilities for each discourse, which we call prob_sequences. Compared to the previous Feedback competition, stacking is much faster since we don't have to deal with a huge amount of candidate sequences. Our features are as follows:
#make features
def get_xgb_features(train_df,prob_sequences):
    features2calculate=[f"instability_{i}" for i in range(4)]+\
    [f"begin_{i}" for i in range(3)]+\
    [f"end_{i}" for i in range(3)]#+\
    #["entropy"]

    calculated_features=[]
    for i,prob_seq in tqdm(enumerate(prob_sequences)):

        tmp=[]
        #quants = np.linspace(0,1,n_quan)
        prob_seq=np.array(prob_seq)
        instability = []
        #all_quants=[]
        tmp.append(np.diff(prob_seq[:,:],0).mean(0))
        tmp.append([(np.diff(prob_seq[:,[1,2]].sum(1))**2).mean()])

        tmp.append(prob_seq[:5,:].mean(0))
        tmp.append(prob_seq[-5:,:].mean(0))

        calculated_features.append(np.concatenate(tmp))


    train_df[features2calculate]=calculated_features
    train_df['len']=[len(s) for s in prob_sequences]

    calculated_features=np.array(calculated_features)
    calculated_features.shape

    p_features=[]
    n_features=[]
    neighbor_features=['Ineffective','Adequate','Effective','discourse_type']
    neighbor_features_values=train_df[neighbor_features].values
    for i in tqdm(range(len(train_df))):
        if i>1 and train_df['essay_id'].iloc[i]==train_df['essay_id'].iloc[i-1]:
            p_features.append(neighbor_features_values[i-1])
        else:
            p_features.append(neighbor_features_values[i])

        if i<(len(train_df)-1) and train_df['essay_id'].iloc[i]==train_df['essay_id'].iloc[i+1]:
            n_features.append(neighbor_features_values[i+1])
        else:
            n_features.append(neighbor_features_values[i])

    train_df[[f+"_previous" for f in neighbor_features]]=p_features
    train_df[[f+"_next" for f in neighbor_features]]=n_features

    train_df['mean_Ineffective']=train_df.groupby("essay_id")["Ineffective"].transform("mean")
    train_df['mean_Adequate']=train_df.groupby("essay_id")["Adequate"].transform("mean")
    train_df['mean_Effective']=train_df.groupby("essay_id")["Effective"].transform("mean")

    train_df['std_Ineffective']=train_df.groupby("essay_id")["Ineffective"].transform("std")
    train_df['std_Adequate']=train_df.groupby("essay_id")["Adequate"].transform("std")
    train_df['std_Effective']=train_df.groupby("essay_id")["Effective"].transform("std")

    train_df['discourse_count']=train_df.groupby("essay_id")['discourse_type'].transform("count")

    cnts=train_df.groupby('essay_id')['discourse_type'].apply(lambda x: x.value_counts())

    #new_df=[]
    discourse_types=['Claim','Evidence','Concluding Statement','Lead','Position','Counterclaim','Rebuttal']
    value_count_hash={}
    for t in discourse_types:
        value_count_hash[t]={}
    for key in cnts.keys():
        value_count_hash[key[1]][key[0]]=cnts[key]

    discourse_cnts=[]    
    for essay_id in train_df['essay_id'].unique():
        row=[essay_id]
        for d in discourse_types:
            try:
                row.append(value_count_hash[d][essay_id])
            except:
                row.append(0)
        discourse_cnts.append(row)

    discourse_cnts=pd.DataFrame(discourse_cnts,columns=['essay_id']+[f'{d}_count' for d in discourse_types])    
    #discourse_cnts

    train_df=train_df.merge(discourse_cnts,how='left',on='essay_id')
    train_df

    #train_df

    return train_df
Since stacking is fast, it works best when we use each fold predictions with xgb separately and then avg. For instance, because I have 6 folds of neural network models and 6 folds of xgb models, this way I have 6x6=36 preds to avg for each single model.
Best single models
All our best single models were deberta-large/deberta-v3-large variants. For Tom and Kkiller, their best single models came from 1st round PL, whereas for me it came from 4th round PL.
Shujun
Tom
Kkiller
Public LB
0.560
0.566
0.562
Private LB
0.558
0.571
0.562
Local CV
0.571
N/A
0.572
Some more tips/tricks
For Tom AWP was useful, and he reported around 0.003 cv improvement with AWP (eps=1e-3 and lr=1e-2 for large models, 1e-4 for xlarge models).
It was also important to split the data with
StratifiedGroupKFold
instead of
GroupKFold
. For me I started out with
GroupKFold
but found better correlation between cv/lb after switching to
StratifiedGroupKFold
.
For ensembling models with the same cv split, we used GP_minimize to find optimal weights and otherwise weights were determined arbitrarily.
Gradient accumulation was useful since we had to deal with very long sequences.