# AI News, AI : Neural Network for beginners (Part 1 of 3)

- On Sunday, September 30, 2018
- By Read More

## AI : Neural Network for beginners (Part 1 of 3)

This article is Part 1 of a series of 3 articles that I am going to post.

The cell body and synapses essentially compute (by a complicated chemical/electrical process) the difference between the incoming excitatory and inhibitory inputs (spatial and temporal summation).

A perceptron models a neuron by taking a weighted sum of inputs and sending the output 1, if the sum is greater than some adjustable threshold value (otherwise it sends 0 - this is the all or nothing spiking described in the biology, see neuron firing section above) also called an activation function.

The inputs (x1,x2,x3..xm) and connection weights (w1,w2,w3..wm) in Figure 4 are typically real values, both postive (+) and negative (-).

The perceptron itself, consists of weights, the summation processor, and an activation function, and an adjustable threshold processor (called bias here after).

Figure 5 Artificial Neuron configuration, with bias as additinal input The bias can be thought of as the propensity (a tendency towards a particular way of behaving) of the perceptron to fire irrespective of its inputs.

Furthermore, if we show to the child new objects that he hasn't seen before, we could expect him to recognize correctly whether the new object is a chair or not, providing that we've given him enough positive and negative examples.

The Perceptron is a single layer neural network whose weights and biases could be trained to produce a correct target vector when presented with the corresponding input vector.

= b + [ T - A ] For all inputs i: W(i) = W(i) + [ T - A ] * P(i)

Where W is the vector of weights, P is the input vector presented to the network, T is the correct result that the neuron should have shown, A is the actual output of the neuron, and b is the bias.

When each epoch (an entire pass through all of the input training vectors is called an epoch) of the training set has occured without error, training is complete.

If a vector, P, not in the training set is presented to the network, the network will tend to exhibit generalization by responding with an output similar to target vectors for input vectors close to the previously unseen input vector P.

Well if we are going to stick to using a single layer neural network, the tasks that can be achieved are different from those that can be achieved by multi-layer neural networks.

As this article is mainly geared towards dealing with single layer networks, let's dicuss those further: Single-layer neural networks (perceptron networks) are networks in which the output unit is independent of the others - each weight effects only one output.

Using perceptron networks it is possible to achieve linear seperability functions like the diagrams shown below (assuming we have a network with 2 inputs and 1 output)

With muti-layer neural networks we can solve non-linear seperable problems such as the XOR problem mentioned above, which is not acheivable using single layer (perceptron) networks.

- On Tuesday, October 2, 2018
- By Read More

In the section on linear classification we computed scores for different visual categories given the image using the formula \( s = W x \), where \(W\) was a matrix and \(x\) was an input column vector containing all pixel data of the image.

In the case of CIFAR-10, \(x\) is a [3072x1] column vector, and \(W\) is a [10x3072] matrix, so that the output scores is a vector of 10 class scores.

There are several choices we could make for the non-linearity (which we’ll study below), but this one is a common choice and simply thresholds all activations that are below zero to zero.

Notice that the non-linearity is critical computationally - if we left it out, the two matrices could be collapsed to a single matrix, and therefore the predicted class scores would again be a linear function of the input.

three-layer neural network could analogously look like \( s = W_3 \max(0, W_2 \max(0, W_1 x)) \), where all of \(W_3, W_2, W_1\) are parameters to be learned.

The area of Neural Networks has originally been primarily inspired by the goal of modeling biological neural systems, but has since diverged and become a matter of engineering and achieving good results in Machine Learning tasks.

Approximately 86 billion neurons can be found in the human nervous system and they are connected with approximately 10^14 - 10^15 synapses.

The idea is that the synaptic strengths (the weights \(w\)) are learnable and control the strength of influence (and its direction: excitory (positive weight) or inhibitory (negative weight)) of one neuron on another.

Based on this rate code interpretation, we model the firing rate of the neuron with an activation function \(f\), which represents the frequency of the spikes along the axon.

Historically, a common choice of activation function is the sigmoid function \(\sigma\), since it takes a real-valued input (the signal strength after the sum) and squashes it to range between 0 and 1.

An example code for forward-propagating a single neuron might look as follows: In other words, each neuron performs a dot product with the input and its weights, adds the bias and applies the non-linearity (or activation function), in this case the sigmoid \(\sigma(x) = 1/(1+e^{-x})\).

As we saw with linear classifiers, a neuron has the capacity to “like” (activation near one) or “dislike” (activation near zero) certain linear regions of its input space.

With this interpretation, we can formulate the cross-entropy loss as we have seen in the Linear Classification section, and optimizing it would lead to a binary Softmax classifier (also known as logistic regression).

The regularization loss in both SVM/Softmax cases could in this biological view be interpreted as gradual forgetting, since it would have the effect of driving all synaptic weights \(w\) towards zero after every parameter update.

The sigmoid non-linearity has the mathematical form \(\sigma(x) = 1 / (1 + e^{-x})\) and is shown in the image above on the left.

The sigmoid function has seen frequent use historically since it has a nice interpretation as the firing rate of a neuron: from not firing at all (0) to fully-saturated firing at an assumed maximum frequency (1).

Also note that the tanh neuron is simply a scaled sigmoid neuron, in particular the following holds: \( \tanh(x) = 2 \sigma(2x) -1 \).

Other types of units have been proposed that do not have the functional form \(f(w^Tx + b)\) where a non-linearity is applied on the dot product between the weights and the data.

TLDR: “What neuron type should I use?” Use the ReLU non-linearity, be careful with your learning rates and possibly monitor the fraction of “dead” units in a network.

For regular neural networks, the most common layer type is the fully-connected layer in which neurons between two adjacent layers are fully pairwise connected, but neurons within a single layer share no connections.

Working with the two example networks in the above picture: To give you some context, modern Convolutional Networks contain on orders of 100 million parameters and are usually made up of approximately 10-20 layers (hence deep learning).

The full forward pass of this 3-layer neural network is then simply three matrix multiplications, interwoven with the application of the activation function: In the above code, W1,W2,W3,b1,b2,b3 are the learnable parameters of the network.

Notice also that instead of having a single input column vector, the variable x could hold an entire batch of training data (where each input example would be a column of x) and then all examples would be efficiently evaluated in parallel.

Neural Networks work well in practice because they compactly express nice, smooth functions that fit well with the statistical properties of data we encounter in practice, and are also easy to learn using our optimization algorithms (e.g.

Similarly, the fact that deeper networks (with multiple hidden layers) can work better than a single-hidden-layer networks is an empirical observation, despite the fact that their representational power is equal.

As an aside, in practice it is often the case that 3-layer neural networks will outperform 2-layer nets, but going even deeper (4,5,6-layer) rarely helps much more.

We could train three separate neural networks, each with one hidden layer of some size and obtain the following classifiers: In the diagram above, we can see that Neural Networks with more neurons can express more complicated functions.

For example, the model with 20 hidden neurons fits all the training data but at the cost of segmenting the space into many disjoint red and green decision regions.

The subtle reason behind this is that smaller networks are harder to train with local methods such as Gradient Descent: It’s clear that their loss functions have relatively few local minima, but it turns out that many of these minima are easier to converge to, and that they are bad (i.e.

Conversely, bigger neural networks contain significantly more local minima, but these minima turn out to be much better in terms of their actual loss.

In practice, what you find is that if you train a small network the final loss can display a good amount of variance - in some cases you get lucky and converge to a good place but in some cases you get trapped in one of the bad minima.

- On Tuesday, October 2, 2018
- By Read More

## For Dummies — The Introduction to Neural Networks we all need ! (Part 1)

This article gives an introduction to perceptrons (single layered neural networks) Update: Part2 of the series is now available for reading here!

Our brain uses the extremely large interconnected network of neurons for information processing and to model the world around us.

The figure depicts a neuron connected with n other neurons and thus receives n inputs (x1, x2, …..

A step function will typically output a 1 if the input is higher than a certain threshold, otherwise it’s output will be 0.

An example would be, Weighing the inputs and adding them together gives, Here, the total input is higher than the threshold and thus the neuron fires.

Furthermore, if the child sees new objects that she hasn’t seen before, we could expect her to recognize correctly whether the new object is a bus or not.

Similarly, input vectors from a training set are presented to the perceptron one after the other and weights are modified according to the following equation, Note: Actually the equation is W(i) = W(i) + a*g’(sum of all inputs)*(T-A)*P(i), where g’ is the derivative of the activation function.

At this time, is an input vector P (already in the training set) is given to the perceptron, it will output the correct value.

The next part of this article series will show how to do this using muti-layer neural networks, using the back propagation training method.

- On Tuesday, October 2, 2018
- By Read More

## Perceptrons: The First Neural Networks

Neural Networks have become incredibly popular over the past few years, and new architectures, neuron types, activation functions, and training techniques pop up all the time in research.

But without a fundamental understanding of neural networks, it can be quite difficult to keep up with the flurry of new work in this area.

To understand the modern approaches, we have to understand the tiniest, most fundamental building block of these so-called deep neural networks: the neuron.

We’ll write Python code (using numpy) to build a perceptron network from scratch and implement the learning algorithm.

To better understand the motivation behind the perceptron, we need a superficial understanding of the structure of biological neurons in our brains.

The point of this cell is to take in some input (in the form of electrical signals in our brains), do some processing, and produce some output (also an electrical signal).

One very important thing to note is that the inputs and outputs are binary (0 or 1)! An individual neuron accepts inputs, usually from other neurons, through its dendrites.

Although the image above doesn’t depict it, the dendrites connect with other neurons through a gap called the synapse that assigns a weight to a particular input.

In other words, if the combination of inputs exceeds a certain threshold, then an output signal is produced, i.e., the neuron “fires.”

If the combination falls short of the threshold, then the neuron doesn’t produce any output, i.e., the neuron “doesn’t fire.”

(We can re-write this as an inner product for succinctness.) There is another term, called the bias, that is just a constant factor.

For mathematical convenience, we can actually incorporate it into our weight vector as and set for all of our inputs.

After taking the weighted sum, we apply an activation function, , to this and produce an activation a.

A white circle means an output of 1 and a black circle means an output of 0, and the axes indicate inputs.

We can create perceptrons that act like gates: they take 2 binary inputs and produce a single binary output!

An intuitive way to understand why perceptrons can only model linearly separable problems is to look the weighted sum equation (with the bias).

(Or, more generally, a hyperplane.) Hence, we’re creating a line and saying that everything on one side of the line belongs to one class and everything on the other side belongs to the other class.

It can be shown that organizing multiple perceptrons into layers and using an intermediate layer, or hidden layer, can solve the XOR problem!

This is a hyperparameter because it is not learned by the perceptron (notice there’s no update rule for !), but we select this parameter.

(For perceptrons, the Perceptron Convergence Theorem says that a perceptron will converge, given that the classes are linearly separable, regardless of the learning rate.

With the update rule in mind, we can create a function to keep applying this update rule until our perceptron can correctly classify all of our inputs.

Before we code the learning algorithm, we need to make some changes to our init function to add the learning rate and number of epochs as inputs.

Technically, if there exists a single weight vector that can separate the classes, there exist an infinite number of weight vectors.

To summarize, perceptrons are the simplest kind of neural network: they take in an input, weight each input, take the sum of weighted inputs, and apply an activation function.

The perceptron learning algorithm fits the intuition by Rosenblatt: inhibit if a neuron fires when it shouldn’t have, and excite if a neuron does not fire when it should have.

We can take that simple principle and create an update rule for our weights to give our perceptron the ability of learning.

- On Monday, September 23, 2019

**Artificial Intelligence - Neurons, Perceptrons, and Neural Networks**

Sound levels rebalanced compared to the last upload, and a small visual tweak made. No difference in script or general animation however. An animated video ...

**What is a Neural Network - Ep. 2 (Deep Learning SIMPLIFIED)**

With plenty of machine learning tools currently available, why would you ever choose an artificial neural network over all the rest? This clip and the next could ...

**3. Some Simple Models of Neurons**

Video from Coursera - University of Toronto - Course: Neural Networks for Machine Learning:

**But what *is* a Neural Network? | Deep learning, chapter 1**

Subscribe to stay notified about new videos: Support more videos like this on Patreon: Or don'

**10.2: Neural Networks: Perceptron Part 1 - The Nature of Code**

In this video, I continue my machine learning series and build a simple Perceptron in Processing (Java). Perceptron Part 2: This ..

**12a: Neural Nets**

NOTE: These videos were recorded in Fall 2015 to update the Neural Nets portion of the class. MIT 6.034 Artificial Intelligence, Fall 2010 View the complete ...

**Lecture 4 | Introduction to Neural Networks**

In Lecture 4 we progress from linear classifiers to fully-connected neural networks. We introduce the backpropagation algorithm for computing gradients and ...

**What is a Neural Network - Ep. 2 (Deep Learning SIMPLIFIED)**

With plenty of machine learning tools currently available, why would you ever choose an artificial neural network over all the rest? This clip and the next could ...

**Lec-4 Nonlinear Activation Units and Learning Mechanisms**

Lecture Series on Neural Networks and Applications by Prof.S. Sengupta, Department of Electronics and Electrical Communication Engineering, IIT Kharagpur.

**Artificial Neural Networks (ANN)**

- A Place for Educational Videos Artificial Neural Networks (ANN) are computers whose architecture is modeled after the brain