# [429th Solution] Google - American Sign Language Fingerspelling Recognition - own, native)

Hello. I had the opportunity to describe my decision and the path by which I came to it) I decided not to deny myself the pleasure of taking this opportunity) Unfortunately, I no longer remember which options I chose for the final submission (I got confused in the list of submitted materials) and it probably doesn't matter considering, that this is a description of the solution that took 429th place. I did a lot of experiments, redid the n-th number of layers, tried to assemble something worthwhile out of them - it helped me become 429) Keras - interesting. Thanks to the organizers for a good time.
Context
Business context:
[https://www.kaggle.com/competitions/asl-fingerspelling/overview](https://www.kaggle.com/competitions/asl-fingerspelling/overview)
Data context:
[https://www.kaggle.com/competitions/asl-fingerspelling/data](https://www.kaggle.com/competitions/asl-fingerspelling/data)
Overview of the Approach
In the beginning, I worked with two publicly available, well-known approaches (thanks to their authors). I was able to achieve cv 0.9+ with an encoder-decoder transformer by adding some LSTM or DepthwiseConv1D+ConvLSTM1D layers before the classifier, but all my attempts to convert and complete the tflite model ended in failure.
The result of this work was a search and an attempt to create a custom LSTM layer. Somewhere on GitHub, I found a variant of a simple LSTM layer (AI Summer link below). Converting it to keras was the start. The created layer did not lead to the desired result, but laid the foundation for an interesting search process. Everything that was - corresponded in Keras. No less amusing was the combination of everything that was, with new finds.
My experiments are based on the work of Rohith Ingilela's and MARK WIJKHUIZEN Loading, data processing, model output, minus my little additions are their activities.
Details of the submission
Once again, I note that I no longer remember which variants were selected by me in the end. I do not exclude that it could even be variants based on the above-mentioned works - pardom. Here are a couple of options for the layers that I used. As I understand it, something similar is often used in work on the classification of poses.
Simple LSTM layer:
:
    def :
        super(Cell_AI_SUMMER, self).
        self.input_length = input_length

        self.linear_forget_w1 = tf.keras.layers.
        self.linear_forget_r1 = tf.keras.layers.
        self.sigmoid_forget = tf.keras.activations.sigmoid

        self.linear_gate_w2 = tf.keras.layers.
        self.linear_gate_r2 = tf.keras.layers.
        self.sigmoid_gate = tf.keras.activations.sigmoid

        self.linear_gate_w3 = tf.keras.layers.
        self.linear_gate_r3 = tf.keras.layers.
        self.activation_gate = tf.keras.activations.tanh

        self.linear_gate_w4 = tf.keras.layers.
        self.linear_gate_r4 = tf.keras.layers.
        self.sigmoid_hidden_out = tf.keras.activations.sigmoid

        self.drop_forget = tf.keras.layers.
        self.activation_final = tf.keras.activations.tanh #gelu #tf.keras.activations.tanh

    def forget(self, x, h):
        x = self.linear
        x = self.drop
        h = self.linear
        h = self.drop
        return self.sigmoid

    def input:
        x_temp = self.linear
        x_temp = self.drop
        h_temp = self.linear
        h_temp = self.drop
        i = self.sigmoid
        return i

    def cell:
        x = self.linear
        h = self.linear
        x = self.drop
        h = self.drop

        k = self.activation
        g = ki

        c = fc_prev
        c_next = g + c
        return c_next

    def out:
        x = self.linear
        h = self.linear
        return self.sigmoid

    def call(self, x, tuple_in ):
        (h, c_prev) = tuple_in
        i = self.input
        f = self.forget(x, h)
        h = tf.cast(h, dtype=tf.float32)
        c_prev = tf.cast(c_prev, dtype=tf.float32)
        c_next = self.cell
        o = self.out
        h_next = oself.activation

        return h_next, c_next
zero_core = 128
(tf.keras.Model):
     ():
        (Sequence2, self).__init__(name=)
        self.input_length = input_length
        self.rnn1 = Cell_AI_SUMMER(, input_length)
        self.rnn2 = Cell_AI_SUMMER(, input_length)

        self.add = tf.keras.layers.Add()
        self.linear = tf.keras.activations.gelu

     ():
        outputs = []
        h_t = tf.zeros([zero_core, self.input_length], dtype=tf.double, name=)
        c_t = tf.zeros([zero_core, self.input_length], dtype=tf.double, name=)
        h_t2 = tf.zeros([zero_core, self.input_length], dtype=tf.double, name=)
        c_t2 = tf.zeros([zero_core, self.input_length], dtype=tf.double, name=)

        h_t, c_t = self.rnn1(, (h_t, c_t))
        h_t2, c_t2 = self.rnn2(h_t, (h_t2, c_t2))

        output = self.linear(h_t2)
        outputs += [output]

         i  (future):
            h_t, c_t = self.rnn1(, (h_t, c_t))
            h_t2, c_t2 = self.rnn2(h_t, (h_t2, c_t2))

            output = self.linear(h_t2)
            outputs += [output]

        outputs = self.add(outputs)
        outputs = self.add([outputs, ])

         outputs
Simple Custom Layer:
:
    def :
        super(Convs, self).
        self.input_length = input_length
        self.ch1, self.ch2, self.ch3 = , ,  #, , 
        self.conv_0 = tf.keras.layers.
        self.conv_1 = tf.keras.layers.
        self.conv_2 = tf.keras.layers.
        self.conv_3 = tf.keras.layers.
        self.conv_4 = tf.keras.layers.
        self.conv_5 = tf.keras.layers.
        self.conv_6 = tf.keras.layers.
        self.conv_7 = tf.keras.layers.
        self.batch1 = tf.keras.layers.
        self.batch2 = tf.keras.layers.
        self.batch3 = tf.keras.layers.
        self.batch4 = tf.keras.layers.
        self.relu =  tf.keras.layers.
        self.add = tf.keras.layers.
        self.drop = tf.keras.layers.
        self.max2 = tf.keras.layers.
        self.uns = tf.keras.layers.
        self.ave = tf.keras.layers.

    def conv0(self, x):
        x = self.conv
        x = self.batch1(x)
        x = self.relu(x)
        x = self.conv
        x = self.drop(x)
        x = self.max2(x)

        return x

    def conv1(self, x):
        x = self.conv
        x = self.batch2(x)
        x = self.relu(x)
        x = self.conv
        x = self.drop(x)
        x = self.uns(x)

        return x

    def conv2(self, x):
        x = self.conv
        x = self.batch3(x)
        x = self.relu(x)
        x = self.conv
        x = self.drop(x)
        x = self.max2(x)

        return x

    def conv3(self, x):
        x = self.conv
        x = self.batch4(x)
        x = self.relu(x)
        x = self.conv
        x = self.drop(x)

        return x

    def call(self, input):
        out = self.conv0(input)
        out = self.conv1(out)
        out = self.conv2(out)
        out = self.conv3(out)
        out = self.add()
        return out
Model example (this is just one of the smaller options):
def get:
    inp = tf.keras.

    x = tf.keras.layers.(inp)

    x = tf.keras.layers.(x)
    x = tf.keras.layers.(x)
    pe = tf.cast(positional, dtype=x.dtype)
    x = x + pe
    x = (x)
    x = (x)
    x1 = (x)
    x = (x)
    x = tf.keras.layers.()
    x = (x)
    x = (x)

    x = tf.keras.layers.(x)

    model = tf.keras.

    return model
Which led to an improvement.
Augmentations at the initial stage. Subsequently, I abandoned them.
Alternating UpSampling1D and MaxPooling1D in custom layers.
Replacing Add with Average.
Leaving in the model a certain amount of Conv1DBlock, TransformerBlock from the 1st place solution of the last competition, by HOYSO48
What I didn't think to do)
Sources
My sincere respect to the authors of the following works:
[https://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference](https://www.kaggle.com/code/markwijkhuizen/aslfr-transformer-training-inference)
[https://github.com/The-AI-Summer/RNN_tutorial/blob/master/cutom_LSTM.py](https://github.com/The-AI-Summer/RNN_tutorial/blob/master/cutom_LSTM.py)
[https://www.kaggle.com/competitions/asl-signs/discussion/406978](https://www.kaggle.com/competitions/asl-signs/discussion/406978)
[https://www.kaggle.com/code/irohith/aslfr-preprocess-dataset-tfrecords-mean-std](https://www.kaggle.com/code/irohith/aslfr-preprocess-dataset-tfrecords-mean-std)
Something like this. Thanks again for an interesting challenge. Have a good day.