# Solution - Single transformer without val dataset

Thanks to the organisers of the PopSign Games and all of the Kaggle community for running such a great competition.
More importantly, my heartfelt thanks go to both
[@markwijkhuizen](https://www.kaggle.com/markwijkhuizen)
and
[@hengck23](https://www.kaggle.com/hengck23)
. MARK contributed a very strong transformer framework, which is the machine learning pipeline for me in this competition. And I will find daily inspiration in
[@hengck23](https://www.kaggle.com/hengck23)
's discussion post.
As a learner, this competition has taken my machine learning skills a step further. Therefore, I would like to
record the results of my experiment here
.
Model Architecture
Shallow networks are the best choice.
Single shallow transformer base (Refer to
[version 5](https://www.kaggle.com/code/markwijkhuizen/gislr-tf-data-processing-transformer-training)
)
Added
label smoothing
.
Used warm up to
lr
.
Remove all LayerNorm.
The main changes I have made that have led to improvements:
Increase
Embedding
to 600.
Add a
Gaussian noise (0.01)
to the FC layer of the Embedding layer.
Increase
epoch
to 230
Deepens the depth of the last FC layer in the transformer layer.
Late Dropout. 0---0.3---0.1
Feature Engineering
Feature engineering was the most difficult part of my entire experiment. I came up with a lot of ideas, but most of them ended up being submission errors.
Here is my summary of options that can bring about an uplift and are feasible.
Use of
lips
,
eyes
(regardless of left or right eye),
main hand
(21 landmarks), and
main pose
(6 landmarks)
Main hand means normalize the right hand to the left hand. The formula is
x = 1 - x
.
The formula of main pose is
x = 1.2 -x
.
Set the input frame count
threshold
to
48
.
Less than that is filled with 0.
More than that is compressed using the
Bilinear Interpolation
method. (I think it only work well to data where the frame rate is a multiple of 48)
Removal of data with only
one frame
.
Add
motion feature
for coordinate X.
Add
Hand distance feature
. (Get inspired by
[@josephzahar](https://www.kaggle.com/josephzahar)
's
[post](https://www.kaggle.com/code/josephzahar/distance-angle-based-features-using-keras)
)
Calculate the Euclidean distance between the discontinuous joints of the hand.
Hence, the dimensionality of each data in my pipeline is 3 dimensions
(x, y, motion)
. For the Hand, it is 213
(x, y, motion, distance)
.
Augmentation
There is no need for mirror(flip) augmentation in this pipeline, as the hand is already normalised in the pre-processing.
In augmentation, I only found rotation to be little useful for me. Specifically, I rotated the data with a small sample size (
data[-42:]
) by 15 degrees up and down. But I think my approach is very poor.
Future Work
How to distinguish similar words, like awake and wake?
Solve the problem of
keras_model_converter.target_spec.supported_types = [tf.float16]
not working for me.
How to do rotation correctly?
Summary of Best Solutions (In processing)
"How To Improve Best Public transformer basded Notebook" from
[@cdeotte](https://www.kaggle.com/cdeotte)
Original Post
[https://www.kaggle.com/competitions/asl-signs/discussion/406302](https://www.kaggle.com/competitions/asl-signs/discussion/406302)
Corresponding Notebook
[https://www.kaggle.com/code/cdeotte/improve-best-public-notebook-lb-0-76](https://www.kaggle.com/code/cdeotte/improve-best-public-notebook-lb-0-76)
Keypoints:
Reduce batch size and increse transformer block.
Ensemble with different model seeds.
All changes
Train 1 model => Train 4 models
Add Time Scale augmentation
Ensemble and apply TFLite FP16 quantization
Change the following parameters:
INPUT_SIZE, 64 => 12
BATCH_ALL_SIGNS_N, 4 => 1
N_EPOCHS, 250 => 120
LANDMARK_UNITS, 384 => 224
UNITS, 512 => 376
NUM_BLOCKS, 2 => 3
MLP_RATIO, 4 => 3
MLP_DROPOUT_RATIO, 0.40 => 0.30
Remove random frame masking
The results of the experiments combined with those tricks
Just use a-i parameters changes in my solution (Step i is necessary, otherwise it would be by gradient messages.).
Public: 0.7525.
Private: 0.8340
Ensemble 2 seed models without quantization. (Surprised by this decline, I am looking into the reasons for it.)
Ensemble Public: 0.7443
Model0 Public: 0.73
Model1 Public: 0.7314
Ensemble Private: 0.8276
Model0 Private: 0.814
Model1 Private: 0.816
I have found the reason. I change
lr
to
0.4*lr
, cause
BATCH_ALL_SIGNS_N
reduces from 4 to 1. This action was wrong, because I use rotation augmentation before training. In other words, all changing variables can affect the choice of
lr
. If those variables stay same but
batch_size
, We need to make the change in
batch_size
proportional to the change in
lr
.
(Answer from this
[post](https://www.kaggle.com/competitions/asl-signs/discussion/406302#2248253)
)
After cancelling the change of
lr
. The result is shown below
Ensemble Public: 0.7683
Ensemble Private: 0.8445
To be resolved
What is Time Scale augmentation? and  How to remove random frame masking?
Why the learning rate is proportional to the batch size?
Answer: This is correct with the other model parameters held constant. However, this is wrong if other variables (epoch, dropout rate, etc.) change.
Where this from "Accurate local validation scheme which estimates LB score" ?
Rule of thumbs
How to choose learning rate
[https://www.kaggle.com/code/isbhargav/guide-to-pytorch-learning-rate-scheduling/notebook](https://www.kaggle.com/code/isbhargav/guide-to-pytorch-learning-rate-scheduling/notebook)
From this
[post](https://www.kaggle.com/competitions/asl-signs/discussion/406302#2248264)
If training requires only a few epochs (i.e. 1, 2, 3, 4, or 5 epochs) like NLP models, I usually just go with constant LR. Then i look at validation metric and if it increases in the last epoch or two, perhaps I will use
LR
for the first epochs and
LR * 0.1
(or 0.3) for the last epochs.
When a model requires many epochs like 20, 40, 80, 160. I usually go with cosine schedule (like this comp). These two techniques work well in most cases.