Updating weights in a neural network

by Vivek p.a   Last Updated July 13, 2019 01:26 AM

I have been trying to code a neural network from scratch and have watched a couple of videos to see how it is implemented.

So I came across this guide that builds a simple neural network in Python.

X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])
y = np.array([[0,1,1,0]]).T
syn0 = 2*np.random.random((3,4)) - 1
syn1 = 2*np.random.random((4,1)) - 1
for j in xrange(60000):
    l1 = 1/(1+np.exp(-(np.dot(X,syn0))))
    l2 = 1/(1+np.exp(-(np.dot(l1,syn1))))
    l2_delta = (y - l2)*(l2*(1-l2))
    l1_delta = l2_delta.dot(syn1.T) * (l1 * (1-l1))
    syn1 += l1.T.dot(l2_delta)
    syn0 += X.T.dot(l1_delta)

I find the last 2 lines confusing shouldn't it be syn1 -= l1.T.dot(l2_delta) and syn0 -= X.T.dot(l1_delta).

I thought that in gradient descent you subtract the slope, but it seems like here it is added. Is this gradient ascent?

Can someone please explain how the last 2 lines work?



Related Questions


Poor accuracy in Tensorflow CNN implementation

Updated January 06, 2018 11:26 AM

How does one debug NaN values in TensorFlow?

Updated February 11, 2018 07:26 AM

Data Shape / Format for RNNs with Multiple Features

Updated April 25, 2017 21:26 PM