Walkthrough

Below is a high level walkthrough of how the code works. For more details, please refer to the documentation in the code.

Phase 1: Forward pass

Step 1: Initilization

First step in a LSTM model is initilizing our short term (h) and long term (C) weights, as well as our input and bias weights to np.zero. In our model, we initilize them both to np.zero values. For the first itteration, the values are randomized, but for each recurring pass, we use the previous output short term and long term weights. We also initilize our grad weights to random values.

 # set up lists to store for short term, long term, and c_tilde (last gate) states at timestamps 0 to t
self.short_term_memory = [np.zeros((self.n_neurons,1)) for x in range(self.max_vector+1)]
self.long_term_memory  = [np.zeros((self.n_neurons,1)) for x in range(self.max_vector+1)]
self.update_gate    = [np.zeros((self.n_neurons,1)) for x in range(self.max_vector)]

# forget gate values
self.dUf = 0.1*np.random.rand(self.n_neurons,1)
self.dbf = 0.1*np.random.rand(self.n_neurons,1)
self.dwf = 0.1*np.random.rand(self.n_neurons,self.n_neurons)

# input gate values
self.dUi = 0.1*np.random.rand(self.n_neurons,1)
self.dbi = 0.1*np.random.rand(self.n_neurons,1)
self.dwi = 0.1*np.random.rand(self.n_neurons,self.n_neurons)

#output gate values
self.dUo = 0.1*np.random.rand(self.n_neurons,1)
self.dbo = 0.1*np.random.rand(self.n_neurons,1)
self.dwo = 0.1*np.random.rand(self.n_neurons,self.n_neurons)

# update gate
self.dUg = 0.1*np.random.rand(self.n_neurons,1)
self.dbg = 0.1*np.random.rand(self.n_neurons,1)
self.dwg = 0.1*np.random.rand(self.n_neurons,self.n_neurons)

Step 2: 'forget' Gate

Once we have our values initilzed, we can begin our first pass. We start by calculating the 'forget gate' which determines what ratio of our previous long term memory we are going to utilize for this pass.

To determine the forget gates outcome, we find the dot product of our input and our short term memory [t] times our input weights and our short term memory weight for at [t] and add or bias:

# forget gate (determine how much of long term memory we will remember or 'forget')
# outout = input * weight + short term * short term weight + forget gate bias
outputf = np.dot(self.Uf, x) + np.dot(self.wf,ht) + self.bf

We then utilize sigmoid as an activation function to map the output of the forget gate to a value between 0 and 1, representing the ratio of our long term memory we are keeping. We save this ratio in our ft variable.

# use our predefined array of sigmoid obje-cts
Sigmf[t].forward(outputf)
#grab output for later
ft = Sigmf[t].output

This value (ft) will be multiplied to our long term memory (C) in the next step.

Step 3: Input Gate

The next step has 2 substeps: (1) calculating what percentage of our new value to add or 'remember' in our long term memory and (2) determining the actual value to add.

We start by calculating same liner combination as our forget gate but with out input gate.

#repeat for input gate
outputi = np.dot(self.Ui, x) + np.dot(self.wi,ht) + self.bi
Sigmi[t].forward(outputi)
it = Sigmi[t].output

This output value from our sigmoid (it) determines the first substeps value (calculating what percentage of our new value to add or 'remember' in our long term memory). The next step requires a different activation function, the TanH, which returns a value between -1 and 1.

Step 4: C_Tilde Gate and Updating Long Term Memory

We use the third gates, or the c_tilde gate value and multiply it with our input gates value to determine what value to add to our long term memory:

# finally repeat for c-tilde
outc_tilde = np.dot(self.Ug,x) + np.dot(self.wg,ht) + self.bg
tan_1[t].forward(outc_tilde)
c_tilde = tan_1[t].output

Now we finally combine our values by multiplying our input gate value by our long term, then adding that to the product of our input gate and our ctilde gate:

ct = np.multiply(ft,ct) + np.multiply(it,c_tilde)

Now out long term memort is updated.

Step 5: Updating Shourt Term Memory

Finally, we update the short term memory to end this iteration and pass the values to the next iteration. We update the Short Term Memory by multiplying the outcome of the output gate by the TanH of the long term memory.

ht = np.multiply(tan_2[t].output,ot)

At this point, all thats left for our forward pass is to update our logs.

# update our short (h) long (c) and ctilde logs
H[t+1] = ht
C[t+1] = ct
C_tilde[t] = c_tilde

#update logs of innter values
F[t] = ft
O[t] = ot
I[t] = it

Step 6:

Since we need to make a decision, we need a set of 2 simple dense layers to give us a decision value. These are simple dense layers with no bells or whistles so I'll save you the redundant explanation. Heres what the forward pass of the dense layers looks like:

 #forward propogation
def forward(self,inputs):
    self.output = np.dot(inputs,self.weights) + self.bias
    self.inputs = inputs

Phase 2: Back Propogation

Here comes the tricky part. Since there are dozens of 'learnable values' and several sets of weights. Back propogation can be scary. A very simple explaination is we are collecting the gradients of the loss for the different gates, performing the chain rule since they are nested, then multiplying the gradients by our learning rates and subtracting that for each weight set. Heres what the implementation looks like.

def backward_pass(self,dvalues):
    # grab our max vector size, short term, and long term memory
    T = self.max_vector
    H = self.short_term_memory
    C = self.long_term_memory
    # our saved inner gate values
    O = self.output_gate
    I = self.input_gate
    C_Tilde = self.update_gate

    # data
    data = self.data
    #activation functions

    sigf = self.Sigf
    sigi = self.Sigi
    sigo = self.Sigo
    tan1 = self.Tan1
    tan2 = self.Tan2

    #BPTT
    dht  = dvalues[-1,:].reshape(self.n_neurons,1)

    for t in reversed(range(T)):
        # working backwards

        # get data in form we can work with
        xt = data[t].reshape(1,1)
        #backwards prop to get dtan2
        tan2[t].backwards(dht)
        #store output of backwards call
        dtanh2 = tan2[t].dinputs

        # dht with respect to tanh
        dhtdtanh = np.multiply(O[t],dtanh2)

        # dct with respect to dft
        dctdft =np.multiply(dhtdtanh,C[t-1])

        #dct with respect to dit
        dctdit = np.multiply(dhtdtanh,C_Tilde[t])

        #dct with respect to dct_tilde
        dctdct_tilde = np.multiply(dhtdtanh,I[t])

        #backwards prop to get dtan1
        tan1[t].backwards(dctdct_tilde)
        #store output of backwards call
        dtanh1 = tan1[t].dinputs


        ...

        #finally, we find derivative of short term memory
        dht = np.dot(self.wf,dsigmf) + np.dot(self.wi,dsigmi)
        +np.dot(self.wo,dsigmo) + np.dot(self.wg,dtanh1)
        +dvalues[t-1,:].reshape(self.n_neurons,1)
    self.short_term_memory = H