{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Isolated Sign Language Recognition with DNN\n\nI will develop a DNN-based isolated sign language recognition model in this notebook.  Because the training records for this dataset contain a variety of frame counts, I will calculate the mean frame for each set of training records to make it simpler to get started and train more quickly. The input shape for this model will be (n, 253, 3) and the output shape (n, 250). It's really difficult to do during inference. In order to do data preparation, such as calculating the mean frame of the test file with input shapes of (None, 253, 3) and imputation, I will build an inference model. This model will have an output shape of (250). \n\nAbout 0.35 CV and 0.33 LB may be obtained using this model. The LB was 0 in the beginning. I surmised that was due to a value being missing. To replace a missing value with 0 and fit my hypothesis, I added the following code to the inference model.\n```python\nx = tf.where(tf.math.is_nan(inputs), tf.zeros_like(inputs), inputs)\n```","metadata":{}},{"cell_type":"markdown","source":"## Configuration","metadata":{}},{"cell_type":"code","source":"class CFG:\n    aggregation_data_path = \"../input/isolated-sign-language-aggregation-dataset/\"\n    data_path = \"../input/asl-signs/\"\n    quick_experiment = False\n    is_training = True\n    use_aggregation_dataset = True\n    num_classes = 250\n    rows_per_frame = 543 ","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:41.260928Z","iopub.execute_input":"2023-05-08T21:49:41.261695Z","iopub.status.idle":"2023-05-08T21:49:41.304317Z","shell.execute_reply.started":"2023-05-08T21:49:41.26166Z","shell.execute_reply":"2023-05-08T21:49:41.303162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Import Packages","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom tqdm.notebook import tqdm\nimport json\nimport os\nimport gc\nfrom sklearn.model_selection import train_test_split","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:41.306425Z","iopub.execute_input":"2023-05-08T21:49:41.307231Z","iopub.status.idle":"2023-05-08T21:49:50.778061Z","shell.execute_reply.started":"2023-05-08T21:49:41.307189Z","shell.execute_reply":"2023-05-08T21:49:50.776959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Utilities","metadata":{}},{"cell_type":"code","source":"def load_relevant_data_subset_with_imputation(pq_path):\n    data_columns = ['x', 'y', 'z']\n    data = pd.read_parquet(pq_path, columns=data_columns)\n    data.replace(np.nan, 0, inplace=True)\n    n_frames = int(len(data) / CFG.rows_per_frame)\n    data = data.values.reshape(n_frames, CFG.rows_per_frame, len(data_columns))\n    return data.astype(np.float32)\n\ndef load_relevant_data_subset(pq_path):\n    data_columns = ['x', 'y', 'z']\n    data = pd.read_parquet(pq_path, columns=data_columns)\n    n_frames = int(len(data) / CFG.rows_per_frame)\n    data = data.values.reshape(n_frames, CFG.rows_per_frame, len(data_columns))\n    return data.astype(np.float32)\n\ndef read_dict(file_path):\n    path = os.path.expanduser(file_path)\n    with open(path, \"r\") as f:\n        dic = json.load(f)\n    return dic","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:50.779823Z","iopub.execute_input":"2023-05-08T21:49:50.78059Z","iopub.status.idle":"2023-05-08T21:49:50.790489Z","shell.execute_reply.started":"2023-05-08T21:49:50.780545Z","shell.execute_reply":"2023-05-08T21:49:50.789296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load data","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(f\"{CFG.aggregation_data_path}train.csv\")\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:50.793546Z","iopub.execute_input":"2023-05-08T21:49:50.793921Z","iopub.status.idle":"2023-05-08T21:49:51.081334Z","shell.execute_reply.started":"2023-05-08T21:49:50.793884Z","shell.execute_reply":"2023-05-08T21:49:51.08016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 21 participants taking part. They each produce between 3,000 and 5,000 training recordings.","metadata":{}},{"cell_type":"code","source":"train.participant_id.nunique()","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:51.083002Z","iopub.execute_input":"2023-05-08T21:49:51.083392Z","iopub.status.idle":"2023-05-08T21:49:51.09708Z","shell.execute_reply.started":"2023-05-08T21:49:51.08336Z","shell.execute_reply":"2023-05-08T21:49:51.095748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.participant_id.value_counts().plot(kind=\"bar\")","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:51.098745Z","iopub.execute_input":"2023-05-08T21:49:51.099043Z","iopub.status.idle":"2023-05-08T21:49:51.444114Z","shell.execute_reply.started":"2023-05-08T21:49:51.099017Z","shell.execute_reply":"2023-05-08T21:49:51.443031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In total, there are 94477 training samples.","metadata":{}},{"cell_type":"code","source":"len(train)","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:51.445924Z","iopub.execute_input":"2023-05-08T21:49:51.446682Z","iopub.status.idle":"2023-05-08T21:49:51.453692Z","shell.execute_reply.started":"2023-05-08T21:49:51.44664Z","shell.execute_reply":"2023-05-08T21:49:51.452623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Sign languages come about 250 different varieties. Each type of sign language has between 300 and 400 examples.","metadata":{}},{"cell_type":"code","source":"label_index = read_dict(f\"{CFG.data_path}sign_to_prediction_index_map.json\")\nindex_label = dict([(label_index[key], key) for key in label_index])\nprint(label_index)\ntrain[\"label\"] = train[\"sign\"].map(lambda sign: label_index[sign])\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:51.455278Z","iopub.execute_input":"2023-05-08T21:49:51.455801Z","iopub.status.idle":"2023-05-08T21:49:51.513032Z","shell.execute_reply.started":"2023-05-08T21:49:51.455761Z","shell.execute_reply":"2023-05-08T21:49:51.511871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"sign\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:51.514507Z","iopub.execute_input":"2023-05-08T21:49:51.515714Z","iopub.status.idle":"2023-05-08T21:49:51.531529Z","shell.execute_reply.started":"2023-05-08T21:49:51.515668Z","shell.execute_reply":"2023-05-08T21:49:51.530467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here are descriptive statistics for number of frames.","metadata":{}},{"cell_type":"code","source":"train.num_frames.describe()","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:51.536088Z","iopub.execute_input":"2023-05-08T21:49:51.536647Z","iopub.status.idle":"2023-05-08T21:49:51.554893Z","shell.execute_reply.started":"2023-05-08T21:49:51.536618Z","shell.execute_reply":"2023-05-08T21:49:51.553695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modeling\nI continue to investigate how to manage this dataset. I utilise mean frame as training input data to make it simple to start with and train more quickly. It takes 30 minutes to load the data. I'll load mean frame data from another dataset that uses the same preprocessing technique to speed up training even further.","metadata":{}},{"cell_type":"code","source":"if CFG.use_aggregation_dataset == False:\n    xs = []\n    ys = []\n    num_frames = np.zeros(len(train))\n    for i in tqdm(range(len(train))):\n        path = f\"{CFG.data_path}{train.iloc[i].path}\"\n        data = load_relevant_data_subset_with_imputation(path)\n        ## Mean Aggregation\n        xs.append(np.mean(data, axis=0))\n        ys.append(train.iloc[i].label)\n        num_frames[i] = data.shape[0]\n        if CFG.quick_experiment and i == 4999:\n            break\n    ## Save number of frames of each training sample for data analysis\n    train[\"num_frames\"] = num_frames\n    X = np.array(xs)\n    y = np.array(ys)\n    print(train[\"num_frames\"].describe())\n    train.to_csv(\"train.csv\", index=False)\nelse:\n    X = np.load(f\"{CFG.aggregation_data_path}X.npy\")\n    y = np.load(f\"{CFG.aggregation_data_path}y.npy\")\nprint(X.shape, y.shape)","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:51.556707Z","iopub.execute_input":"2023-05-08T21:49:51.557117Z","iopub.status.idle":"2023-05-08T21:49:56.997552Z","shell.execute_reply.started":"2023-05-08T21:49:51.557056Z","shell.execute_reply":"2023-05-08T21:49:56.996316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_model():\n    inputs = tf.keras.Input((543, 3), dtype=tf.float32)\n    vector = tf.keras.layers.Dense(128, activation=\"swish\")(inputs)\n    vector = tf.keras.layers.Dense(64, activation=\"swish\")(vector)\n    vector = tf.keras.layers.Dense(32, activation=\"swish\")(vector)\n    vector = tf.keras.layers.Dense(16, activation=\"swish\")(vector)\n    vector = tf.keras.layers.Dropout(0.1)(vector)\n    vector = tf.keras.layers.Flatten()(vector)\n    output = tf.keras.layers.Dense(250, activation=\"softmax\")(vector)\n    model = tf.keras.Model(inputs=inputs, outputs=output)\n    model.compile(\n        loss=tf.keras.losses.SparseCategoricalCrossentropy(), \n        metrics=[\n            \"accuracy\", \n            tf.keras.metrics.SparseTopKCategoricalAccuracy(k=5, name=\"top-5-accuracy\"),\n            tf.keras.metrics.SparseTopKCategoricalAccuracy(k=10, name=\"top-10-accuracy\")\n        ]\n    )\n    return model","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:57.000305Z","iopub.execute_input":"2023-05-08T21:49:57.001076Z","iopub.status.idle":"2023-05-08T21:49:57.010811Z","shell.execute_reply.started":"2023-05-08T21:49:57.001032Z","shell.execute_reply":"2023-05-08T21:49:57.009451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_name = \"model.h5\"\nif CFG.is_training:\n    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)\n    print(X_train.shape, y_train.shape, X_val.shape, y_val.shape)\n    del X, y\n    gc.collect()\n    model = get_model()\n    callbacks = [\n        tf.keras.callbacks.ModelCheckpoint(model_name, save_best_only=True, restore_best_weights=True, monitor=\"val_accuracy\", mode=\"max\")\n    ]\n    model.fit(X_train, y_train, epochs=50, validation_data=(X_val, y_val), batch_size=128, callbacks=callbacks)\n    model.load_weights(model_name)\nelse:\n    model = tf.keras.models.load_model(f\"../input/sign-language-prediction-model/{model_name}\")\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:49:57.012552Z","iopub.execute_input":"2023-05-08T21:49:57.012956Z","iopub.status.idle":"2023-05-08T21:57:24.72861Z","shell.execute_reply.started":"2023-05-08T21:49:57.012917Z","shell.execute_reply":"2023-05-08T21:57:24.727744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create an inference model\nThis inference model wraps the previous trained DNN model and do following preprocessing:\n* Replace nan value with 0\n* calcuate mean frame of the input tensor with (None, 543, 3) shape and convert to (1, 543, 3) shape.","metadata":{}},{"cell_type":"code","source":"def get_inference_model(model):\n    inputs = tf.keras.Input((543, 3), dtype=tf.float32, name=\"inputs\")\n    x = tf.where(tf.math.is_nan(inputs), tf.zeros_like(inputs), inputs)\n    x = tf.reduce_mean(x, axis=0, keepdims=True)\n    x = model(x)\n    output = tf.keras.layers.Activation(activation=\"linear\", name=\"outputs\")(x)\n    inference_model = tf.keras.Model(inputs=inputs, outputs=output) \n    inference_model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(), metrics=[\"accuracy\"])\n    return inference_model","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:57:24.729734Z","iopub.execute_input":"2023-05-08T21:57:24.730111Z","iopub.status.idle":"2023-05-08T21:57:24.741931Z","shell.execute_reply.started":"2023-05-08T21:57:24.73006Z","shell.execute_reply":"2023-05-08T21:57:24.741069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inference_model = get_inference_model(model)\ninference_model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:57:24.743165Z","iopub.execute_input":"2023-05-08T21:57:24.744264Z","iopub.status.idle":"2023-05-08T21:57:24.845778Z","shell.execute_reply.started":"2023-05-08T21:57:24.744226Z","shell.execute_reply":"2023-05-08T21:57:24.844881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Converter\n Convert the kereas to tensorflow lite model","metadata":{}},{"cell_type":"code","source":"converter = tf.lite.TFLiteConverter.from_keras_model(inference_model)\ntflite_model = converter.convert()\nmodel_path = \"model.tflite\"\n# Save the model.\nwith open(model_path, 'wb') as f:\n    f.write(tflite_model)","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:57:24.846955Z","iopub.execute_input":"2023-05-08T21:57:24.847346Z","iopub.status.idle":"2023-05-08T21:57:28.872227Z","shell.execute_reply.started":"2023-05-08T21:57:24.847304Z","shell.execute_reply":"2023-05-08T21:57:28.87087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Making inferences\nLet's utilise TFLite to draw conclusions about our file in the methods described below. I'll draw conclusions using 10,000 samples to assess the speed of inference. At least ten inferences per second should be achievable with the adapted TFLite model.","metadata":{}},{"cell_type":"code","source":"!pip install tflite-runtime","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:57:28.873994Z","iopub.execute_input":"2023-05-08T21:57:28.874396Z","iopub.status.idle":"2023-05-08T21:57:40.397217Z","shell.execute_reply.started":"2023-05-08T21:57:28.874352Z","shell.execute_reply":"2023-05-08T21:57:40.395571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tflite_runtime.interpreter as tflite\ninterpreter = tflite.Interpreter(model_path)\nfound_signatures = list(interpreter.get_signature_list().keys())\nprediction_fn = interpreter.get_signature_runner(\"serving_default\")\nfor i in tqdm(range(10000)):\n    frames = load_relevant_data_subset(f'/kaggle/input/asl-signs/{train.iloc[i].path}')\n    output = prediction_fn(inputs=frames)\n    sign = np.argmax(output[\"outputs\"])\n    if i % 100 == 0:\n        print(f\"Predicted Label: {index_label[sign]}, Actual Label: {train.iloc[i].sign}\")","metadata":{"execution":{"iopub.status.busy":"2023-05-08T21:57:40.399497Z","iopub.execute_input":"2023-05-08T21:57:40.399946Z","iopub.status.idle":"2023-05-08T22:01:34.093857Z","shell.execute_reply.started":"2023-05-08T21:57:40.399898Z","shell.execute_reply":"2023-05-08T22:01:34.092616Z"},"trusted":true},"execution_count":null,"outputs":[]}]}