{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n#
Introduction to LSTMs
\n\n\n### What will you learn?\n\n- The theoretical background on how LSTMs work.\n- Learn the different components of the LSTM network architecture, like LSTM layers, cells and gates.\n\nThis notebook was inspired by Michael Phi's [Illustrated Guide to LSTM's and GRU's: A step by step explanation](https://www.youtube.com/watch?v=8HyCNIVRbSU) so support him by giving the thumb's up πŸ‘πŸΌ!\n\n### Table of Contents\n\nThe notebook will be divided into separate sections to provide an organized walk-through. The sections are:\n\n
\n
    \n
  1. Introduction
    2. \n
  2. LSTM Layers and LSTM Cells
    3. \n \n
  3. States
    4. \n \n
  4. Gates
    5. \n \n
  5. Import Libraries
    6. \n
  6. LSTM Layer PyTorch
    7. \n
  7. Minimal Example: Feeding 2 Sentences to an LSTM
    8. \n \n
  8. References
    9. \n
\n
","metadata":{}},{"cell_type":"markdown","source":"# 1 | Introduction [↑](#top) \n***\n\nLSTM networks were first proposed by Sepp Hochreiter and JΓΌrgen Schmidhuber in 1997 as a solution to the vanishing gradient problem in standard recurrent neural networks (RNNs). The vanishing gradient problem occurs when the gradients used to update the weights of a neural network become very small, making it difficult for the network to learn long-term dependencies in the input data.\n\nAn LSTM network is a type of neural network that is specifically designed to process sequences of data, such as time series or natural language. The network is composed of multiple LSTM layers, which are stacked on top of one another to form a deep neural network.\n\nEach LSTM layer is composed of a number of LSTM cells. An LSTM cell is a single processing unit within an LSTM layer that is capable of storing information over time. It consists of several components, including a memory cell, an input gate, a forget gate, and an output gate.\n\n$$ LSTM\\:Network \\: \\xrightarrow{\\text{composed of}} \\: LSTM\\:Layers \\: \\xrightarrow{\\text{composed of}} \\: LSTM\\:Cells \\: \\xrightarrow{\\text{composed of}} Gates$$\n\n\nThis is a basic diagram of the LSTM architecture, we will come back to it many times throughout this notebook:\n\n
\n\nThe basic idea behind LSTMs is to introduce a set of memory cells that can store information for an extended period of time. Each memory cell is equipped with three gates - an input gate, an output gate, and a forget gate - which control the flow of information into and out of the cell.\n\nThe input gate decides which information to add to the memory cell, while the forget gate determines which information to remove from the cell. The output gate controls the flow of information out of the cell and into the next layer of the network.\n\nBy allowing the network to selectively retain or discard information over time, LSTMs are able to learn long-term dependencies in the input data more effectively than traditional RNNs.\n\nOver the years, LSTMs have become an increasingly popular choice for a wide range of applications, including speech recognition, natural language processing, and time series analysis. They have also been extended and modified in various ways, such as adding peephole connections, using different activation functions, and introducing multiple layers.\n\nToday, LSTMs remain one of the most widely used types of neural networks, and they continue to be an active area of research in the field of deep learning.\n\n","metadata":{}},{"cell_type":"code","source":"from IPython.display import YouTubeVideo\n\nYouTubeVideo('8HyCNIVRbSU', width=800, height=300)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T22:05:01.765906Z","iopub.execute_input":"2023-03-21T22:05:01.767267Z","iopub.status.idle":"2023-03-21T22:05:01.831865Z","shell.execute_reply.started":"2023-03-21T22:05:01.767219Z","shell.execute_reply":"2023-03-21T22:05:01.830402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2 | LSTM Layers and LSTM Cells [↑](#top) \n***\n\n### LSTM Layer[↑](#top)\n\nAn **LSTM layer** (Long Short-Term Memory) is a type of recurrent neural network (RNN) layer that is commonly used in deep learning for sequence modeling tasks, such as language modeling, speech recognition, and time series prediction. An LSTM layer contains multiple LSTM cells, which are the basic building blocks of the layer. \n\nThe image represents an **LSTM Neural Network**, the **green blocks** are **LSTM Cells** (except of course for the FC layers), while the groups of these cells with the same number are the **LSTM Layers**. For example, **all LSTM cells numbered** 1 compose the **LSTM Layer** 1. This network also has some Fully Connected Layers (FCL) in the end. So, in the image:\n1. There are *T* number of LSTM cells per layer.\n2. The are *2* LSTM layers.\n2. The are *2* Fully Connected layers.\n\n
\n\n### LSTM Cell[↑](#top)\n\nAn **LSTM cell** is a specific type of RNN cell that is designed to address the vanishing gradient problem that can occur when training traditional RNNs on long sequences of data. The LSTM cell is composed of several gates, which are used to control the flow of information through the cell. The gates include the input gate, forget gate, and output gate, each of which serves a specific purpose in determining which information is passed through the cell.\n\n
\n\nSo, the main difference between an LSTM layer and an LSTM cell is that the layer is a higher-level structure that contains multiple cells, while the cell is the basic unit that processes the sequence data and controls the flow of information.\n\n### Similarities with Linear Layers[↑](#top)\n\nThere are some similarities between the relationship of LSTM cells to LSTM layers and the relationship of Neurons to Linear layers, but the two relationships are not exactly the same.\n\nIn both cases, the lower-level building blocks (LSTM cells or neurons) perform a specific function on the input data, and the higher-level structures (LSTM layers or linear layers) combine the outputs of the lower-level building blocks to produce a final output. So in a certain way or analogy you could think that \"neurons are to linear layers what LSTM cells are to LSTM layers\".\n\nHowever, there are some important differences. Firstly, LSTM cells are specifically designed for processing sequential data, while neurons in a linear layer can be used for processing any kind of data. Secondly, while LSTM cells have a complex structure with multiple internal components (such as input gates, output gates, and memory cells) that allow them to handle long-term dependencies in sequential data, neurons in a linear layer typically have a simpler structure with only one or a few learnable parameters.\n\nSo while LSTM cells and neurons both serve as the basic building blocks of their respective layers, the specific functions they perform and the complexity of their internal structure are different.","metadata":{}},{"cell_type":"markdown","source":"# 3 | States [↑](#top) \n***\n\n\n\n\n\n### Cell State [↑](#top) \n\nThe **cell states** act as a \"transport highway\" that transfers relative information all the way down to the sequence of LSTM cells. We can think about it as the memory of the network. Because the cell states can carry information throughout the sequence of LSTM cells in theory even information from earlier time steps could be carried all the way to the last time step thus reducing the effects of short-term memory.\n\n
\n
\n\nThe cell state is calculated by the following formula:\n$$c_{t} = f_{t} \\ast c_{t-1} + i_{t} \\ast g_{t}$$\n\nWhere:\n\n- $f_{t}$ is the forget gate output.\n- $c_{t-1}$ is the previous cell state.\n- $i_{t}$ is the input gate output.\n- $g_{t}$ is the cell gate output.\n\n### Hidden State [↑](#top) \n\nThe **hidden state** is a vector of values that is updated at each time step as the network processes input sequences.\n\nThe hidden state is used to capture and store information about the input sequence that has been processed so far, and this information is passed along to the next time step. The hidden state serves as a kind of \"memory\" for the network, allowing it to keep track of important features of the input sequence as it processes it.\n\nThe LSTM architecture is designed to allow the network to selectively remember or forget information in the hidden state using special gating mechanisms. These gates allow the network to control the flow of information through the hidden state, which can help to prevent the network from overfitting or forgetting important features of the input sequence.","metadata":{}},{"cell_type":"markdown","source":"# 4 | Gates [↑](#top) \n***\n\nSo we have three different types of gates that regulate information flow in an LSTM cell:\n1. Forget Gate\n2. Input Gate\n3. Output Gate\n\n
\n\n\n### Forget Gate[↑](#top)\n\nThe **forget gate** decides what information should be thrown or kept away. Information from a previous hidden state $h_{t-1}$ and the information from the current input $x_{t}$ are concatenated and passed through the sigmoid function. Values come out between 0 and 1. The closer to 0 means \"forget\" and a closer to 1 means \"keep\" \n\n
\n
\n\n### Input Gate[↑](#top)\n\nTo update the cell state we have the **input gate**. The input gate consists of two parts: one with a sigmoid function and another one with a hyperbolic tangent function (usually called the cell gate). Just as with the forget gate, both are input the concatenation of $h_{t-1}$ and $x_{t}$. The sigmoid part is exactly as the forget gate, the tanh part squish values between -1 and 1 which helps to regulate the network. Both parts are multiplied which in essence means that the sigmoid output decides which information is important to keep from the tanh output.\n\n
\n
\n\n### Output Gate[↑](#top)\n\nThe **output gate** decides what the next hidden state should be. First $h_{t-1}$ and $x_{t}$ are passed once again through a sigmoid function. Then the current cell state is passed through a hyperbolic tangent function. We then multiply the sigmoid output with the tanh output. This decides which information from the cell state the hidden state should carry. \n\n
\n
\n\nThe LSTM outputs the next time step's input cell state an input hidden state.","metadata":{}},{"cell_type":"markdown","source":"# 5 | Import Libraries [↑](#top) \n***","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport os\nimport torch\nimport torch.nn as nn","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-03-22T15:39:23.827586Z","iopub.execute_input":"2023-03-22T15:39:23.828019Z","iopub.status.idle":"2023-03-22T15:39:25.084862Z","shell.execute_reply.started":"2023-03-22T15:39:23.827973Z","shell.execute_reply":"2023-03-22T15:39:25.083506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6 | LSTM Layer PyTorch [↑](#top) \n***\n\n
","metadata":{}},{"cell_type":"code","source":"BATCH_SIZE = 8 # Number of sequences in your batch\nHIDDEN_SIZE = 16 # Number of features in the hidden state \nINPUT_SIZE = 384 # Size of each embedding in the sequence\nNUM_LAYERS = 2 # Number of LSTM layers in the LSTM network\nSEQUENCE_LENGTH = 50 # Number of embeddings per sequence\n\n# === LSTM Network ===\nLSTM = nn.LSTM(\n input_size=INPUT_SIZE,\n hidden_size=HIDDEN_SIZE, \n num_layers=NUM_LAYERS, \n batch_first=True\n)\n\ninputs = torch.randn(BATCH_SIZE, SEQUENCE_LENGTH, INPUT_SIZE) # LSTM Network input\nh0 = torch.randn(NUM_LAYERS, BATCH_SIZE, HIDDEN_SIZE) # Initial Hidden state\nc0 = torch.randn(NUM_LAYERS, BATCH_SIZE, HIDDEN_SIZE) # Initial Cell state\noutputs, (hn, cn) = LSTM(inputs, (h0, c0)) # LSTM Network output\n\nprint(f\"Input batch shape: {inputs.shape}\")\nprint(f\"Hidden state 0 shape: {h0.shape}\")\nprint(f\"Cell state 0 shape: {c0.shape}\")\nprint(f\"Output batch shape: {outputs.shape}\")","metadata":{"execution":{"iopub.status.busy":"2023-03-22T16:31:44.482417Z","iopub.execute_input":"2023-03-22T16:31:44.483609Z","iopub.status.idle":"2023-03-22T16:31:44.509277Z","shell.execute_reply.started":"2023-03-22T16:31:44.483546Z","shell.execute_reply":"2023-03-22T16:31:44.507982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7 | Minimal Example: Feeding 2 Sentences to an LSTM [↑](#top) \n***\n\nThe author of this example is @chrisvdweth. You can find the source [code](https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/notebooks/minimal-example-lstm-input.ipynb) in his GitHub.\n\n\n### Create vocabulary[↑](#top)","metadata":{}},{"cell_type":"code","source":"sentences = ['i like fresh bread', 'i hate stale bread']\n\nvocabulary = set()\nfor sent in sentences:\n for word in sent.split():\n vocabulary.add(word)\n\nprint(vocabulary)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T17:06:41.975099Z","iopub.execute_input":"2023-03-22T17:06:41.975534Z","iopub.status.idle":"2023-03-22T17:06:41.982707Z","shell.execute_reply.started":"2023-03-22T17:06:41.975499Z","shell.execute_reply":"2023-03-22T17:06:41.981334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create indices to vectorize words[↑](#top)","metadata":{}},{"cell_type":"code","source":"# Most NLP tasks require some basic tokens, e.g., for padding, unknown words, start of sequence, end of sequence\nword2index = {'': 0, '': 1, '': 2, '': 3}\n\nfor word in vocabulary:\n word2index[word] = len(word2index)\n \nprint(word2index)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T17:06:44.987273Z","iopub.execute_input":"2023-03-22T17:06:44.987673Z","iopub.status.idle":"2023-03-22T17:06:44.994586Z","shell.execute_reply.started":"2023-03-22T17:06:44.987638Z","shell.execute_reply":"2023-03-22T17:06:44.993265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Vectorize sentences[↑](#top)","metadata":{}},{"cell_type":"code","source":"batch = [[word2index[word] for word in sent.split()] for sent in sentences]\n\nprint(batch)\nprint()\n\n# Let's make a numpy array out of it\nbatch = np.array(batch)\n\nprint(batch)\nprint('The shape of batch is:', batch.shape)\nprint()\n\n# Let's make a PyTorch tensor out of it\nbatch = torch.tensor(batch, dtype=torch.long)\nprint(batch)\nprint('The shape of batch is:', batch.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T17:06:45.789212Z","iopub.execute_input":"2023-03-22T17:06:45.789643Z","iopub.status.idle":"2023-03-22T17:06:45.814179Z","shell.execute_reply.started":"2023-03-22T17:06:45.789604Z","shell.execute_reply":"2023-03-22T17:06:45.812579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define Embedding layer[↑](#top)\n\nThe Embedding layer is like a look-up table. You can imagine it like a dictionary, where the keys are the words in our vocabulary and the values are embeddings. \n\nAn embedding in deep learning refers to a process of mapping an input object, such as a word, a categorical variable or even an image, to a continuous **vector** space, usually with a lower dimensionality.\n\nEmbeddings are widely used in natural language processing (NLP) tasks to represent words or phrases in a way that **captures their semantic relationships**. In such cases, the embedding is typically learned by training a neural network on a large corpus of text data.\n\nSo in simple terms, embeddings are numerical vectors which represent an object and capture the characteristics and relationships of these original objects. For example, the embedding of the word *king* plus the embedding of the word *man* minus the embedding of the word *woman* gives you the embedding of the word *queen*.\n\n$$e_{queen} \\approx e_{king} + e_{man} - e_{woman}$$\n\nHowever, each word embedding in this example is a random vector which has no semantic meaning.","metadata":{}},{"cell_type":"code","source":"vocab_size = len(word2index) # vocab_size reflects all known words in the index\nembed_dim = 10 # Let's assume the word embeddings are of size 10 to keep it simple\nword_embedding_layer = nn.Embedding(vocab_size, embed_dim)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T17:06:46.608761Z","iopub.execute_input":"2023-03-22T17:06:46.609214Z","iopub.status.idle":"2023-03-22T17:06:46.616686Z","shell.execute_reply.started":"2023-03-22T17:06:46.609173Z","shell.execute_reply":"2023-03-22T17:06:46.615088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Push batch through Embedding layer[↑](#top)","metadata":{}},{"cell_type":"code","source":"batch = word_embedding_layer(batch)\n\nprint('The shape of batch is:', batch.shape)\nprint()\nprint(batch)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T17:06:47.478388Z","iopub.execute_input":"2023-03-22T17:06:47.478803Z","iopub.status.idle":"2023-03-22T17:06:47.524963Z","shell.execute_reply.started":"2023-03-22T17:06:47.478767Z","shell.execute_reply":"2023-03-22T17:06:47.524069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Define LSTM layer[↑](#top)","metadata":{}},{"cell_type":"code","source":"hidden_dim = 32 # Let's set the size if the hidden dimension to 32\n\nlstm = nn.LSTM(embed_dim,\n hidden_dim,\n batch_first=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T17:06:49.524257Z","iopub.execute_input":"2023-03-22T17:06:49.525370Z","iopub.status.idle":"2023-03-22T17:06:49.531836Z","shell.execute_reply.started":"2023-03-22T17:06:49.525310Z","shell.execute_reply":"2023-03-22T17:06:49.530312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Push batch through LSTM layer[↑](#top)","metadata":{}},{"cell_type":"code","source":"# Initialize hidden state (here with zeros)\nbatch_size = batch.shape[0]\n(h, c) = (torch.zeros(1, batch_size, hidden_dim), torch.zeros(1, batch_size, hidden_dim)) \n\nlstm_out, (h, c) = lstm(batch, (h, c))\n\nprint('The shape of lstm_out is:', lstm_out.shape) # (batch_size, seq_len, hidden_dim)\nprint('The shape of h is:', h.shape) # (num_layers*num_directions, batch_size, hidden_dim)","metadata":{"execution":{"iopub.status.busy":"2023-03-22T17:06:50.685379Z","iopub.execute_input":"2023-03-22T17:06:50.685781Z","iopub.status.idle":"2023-03-22T17:06:50.694447Z","shell.execute_reply.started":"2023-03-22T17:06:50.685748Z","shell.execute_reply":"2023-03-22T17:06:50.692966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | References [↑](#top) \n***\n\nTo get a better understanding of LSTMs these are some good readings and videos:\n1. [PyTorch documentation on LSTM Cell](https://pytorch.org/docs/stable/generated/torch.nn.LSTMCell.html#lstmcell)\n2. [PyTorch documentation on LSTM Layer](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM)\n2. [Illustrated Guide to LSTM's and GRU's](https://www.youtube.com/watch?v=8HyCNIVRbSU)\n3. [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)\n4. [Implementation Differences in LSTM Layers](https://towardsdatascience.com/implementation-differences-in-lstm-layers-tensorflow-vs-pytorch-77a31d742f74#:~:text=The%20output%20of%20the%20Pytorch,another%20tuple%20with%20two%20elements.)","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}