BI Corner

The intent of this article is not to tell you everything you wanted to know about artificial neural networks (ANN) and were afraid to ask.

It is used is primarily with functions dealing with regression analyses like linear models (lm) and general linear models (glm).

As essential arguments, we must specify a formula in terms of response variables ~ sum of covariates and a data set containing covariates and response variables.

This data set contains data of a case-control study that investigated infertility after spontaneous and induced abortion (Trichopoulos et al., 1976).

The data set consists of 248 observations, 83 women, who were infertile (cases), and 165 women, who were not infertile (controls).

 Both variables take possible values 0, 1, and 2 relating to 0, 1, and 2 or more prior abortions.

The function neuralnet, used for training a neural network, provides the opportunity to define the required number of hidden layers and hidden neurons according to the needed complexity.

The most important arguments of the function are the following: The usage of neuralnet is described by modeling the relationship between the case-control status (case) as response variable and the four covariates age, parity, induced and spontaneous.

neuralnet(formula = case~age+parity+induced+spontaneous, data = infert, hidden = 2, err.fct = 'ce', linear.output = FALSE)

2.623245311046 The training  process needed 5254 steps until  all absolute partial  derivatives  of  the  error  function were smaller than 0.01 (the default threshold).

For instance, the intercepts of the first hidden layer are 5.59 and 1.06 and the four  weights leading to the first hidden neuron are estimated as −0.12, 1.77, −2.20, and −3.37  for the covariates age,  parity,  induced and spontaneous, respectively.

36       4        2            1      0.4904344475 In this case, the object nn$net.result is a list consisting of only one element relating to one calculated replication.

nn.bp nn.bp Call:neuralnet(formula = case~age+parity+induced+spontaneous, data = infert, hidden = 2, learningrate = 0.01, algorithm =  'backprop', err.fct = 'ce', linear.output = FALSE)

The generalized weight w̃i is defined as the contribution of the ith covariate to the log-odds: The generalized weight expresses the effect of each covariate xi and thus has an analogous interpretation as the ith regression parameter in regression models.

 0.1360035 -2.0427123 2.5449249 3.8975730 The columns refer to the four covariates age (j =1), parity (j = 2), induced (j = 3), and spontaneous(j = 4) and a generalized weight is given for each observation even though they are equal for each covariate combination.

 The plot includes by default the trained synaptic weights, all intercepts as well as basic information about the training process like the overall error and the number of steps needed to converge.

The second possibility to visualize the results is to plot generalized weights.  gwplot uses the calculated generalized weights provided by nn$generalized.weights and can be used by the following statements: >

The distribution of the generalized weights suggests that the covariate age has no effect on the case-control status since all generalized weights are nearly zero and that at least the two covariates induced and spontaneous have a non-linear effect since the variance of their generalized weights is overall greater than one.

It learns an approximation of the relationship between inputs and outputs and can then be used to predict outputs o(xnew ) relating to new covariate combinations xnew .

To  stay  with the  example,  predicted  outputs can be calculated  for  instance for  missing  combinations  with age=22, parity=1, induced ≤ 1, and  spontaneous ≤ 1.  They are provided by new.output$net.result >

Since the covariate age has no effect on the outcome and the related neuron is thus irrelevant, a new neural network  (,  which has only the three input variables parity,  induced,  and spontaneous, has to  be  trained  to  demonstrate  the  usage of confidence.interval.

Call: neuralnet(formula = case ~ parity + induced + spontaneous, data = infert,     hidden = 2, err.fct = 'ce', linear.output = FALSE)

Neural networks: further insights into error function, generalized weights and others

With the help of the neuralnet() function contained in neuralnet package, the training of NN model is extremely easy (1).

After model training, the topology of the NN can be visualized using the generic function plot() with many options for adjusting appearance of the plot.

For example, a vector c(4,2,5) indicates a neural network with three hidden layers, and the numbers of neurons for the first, second and third layers are 4, 2 and 5, respectively.

Activation function transforms aggregated input signals, also known as induced local field, into output signal (3).

and it can be expressed as: where l=1,2,3,…,L indexes observations, h=1,2,…,H is the output nodes, and o is the predicted output and y is the observed output.

However, the comprehension of mathematical expression of cross entropy is a little more challenging: overall, the error function describes the deviation of predicted outcomes from the observed ones.

Absolute partial derivatives of the error function with respect to weight (∂ E/∂ w) are slopes used to guide us to find a minimum error (e.g., a slope of zero indicates the nadir).

In traditional backpropagation, the learning rate is fixed, but it can be changed during training process in resilient backpropagation (5,6).

Weight update of resilient backpropagation in each iteration is written in the following equation: where the learning rate can be changed during training process according to the sign of the partial derivative.

The derivative of error function is negative at step t, then the next weight should be greater than in order to find a weight with a slope equal or close to zero.

By default, the neuralnet() function uses 0.01 as the threshold for partial derivative of error function to stop iteration.

In the section on linear classification we computed scores for different visual categories given the image using the formula \( s = W x \), where \(W\) was a matrix and \(x\) was an input column vector containing all pixel data of the image.

In the case of CIFAR-10, \(x\) is a [3072x1] column vector, and \(W\) is a [10x3072] matrix, so that the output scores is a vector of 10 class scores.

There are several choices we could make for the non-linearity (which we’ll study below), but this one is a common choice and simply thresholds all activations that are below zero to zero.

Notice that the non-linearity is critical computationally - if we left it out, the two matrices could be collapsed to a single matrix, and therefore the predicted class scores would again be a linear function of the input.

three-layer neural network could analogously look like \( s = W_3 \max(0, W_2 \max(0, W_1 x)) \), where all of \(W_3, W_2, W_1\) are parameters to be learned.

The area of Neural Networks has originally been primarily inspired by the goal of modeling biological neural systems, but has since diverged and become a matter of engineering and achieving good results in Machine Learning tasks.

Approximately 86 billion neurons can be found in the human nervous system and they are connected with approximately 10^14 - 10^15 synapses.

The idea is that the synaptic strengths (the weights \(w\)) are learnable and control the strength of influence (and its direction: excitory (positive weight) or inhibitory (negative weight)) of one neuron on another.

Based on this rate code interpretation, we model the firing rate of the neuron with an activation function \(f\), which represents the frequency of the spikes along the axon.

Historically, a common choice of activation function is the sigmoid function \(\sigma\), since it takes a real-valued input (the signal strength after the sum) and squashes it to range between 0 and 1.

An example code for forward-propagating a single neuron might look as follows: In other words, each neuron performs a dot product with the input and its weights, adds the bias and applies the non-linearity (or activation function), in this case the sigmoid \(\sigma(x) = 1/(1+e^{-x})\).

As we saw with linear classifiers, a neuron has the capacity to “like” (activation near one) or “dislike” (activation near zero) certain linear regions of its input space.

With this interpretation, we can formulate the cross-entropy loss as we have seen in the Linear Classification section, and optimizing it would lead to a binary Softmax classifier (also known as logistic regression).

The regularization loss in both SVM/Softmax cases could in this biological view be interpreted as gradual forgetting, since it would have the effect of driving all synaptic weights \(w\) towards zero after every parameter update.

The sigmoid non-linearity has the mathematical form \(\sigma(x) = 1 / (1 + e^{-x})\) and is shown in the image above on the left.

The sigmoid function has seen frequent use historically since it has a nice interpretation as the firing rate of a neuron: from not firing at all (0) to fully-saturated firing at an assumed maximum frequency (1).

Also note that the tanh neuron is simply a scaled sigmoid neuron, in particular the following holds: \( \tanh(x) = 2 \sigma(2x) -1 \).

Other types of units have been proposed that do not have the functional form \(f(w^Tx + b)\) where a non-linearity is applied on the dot product between the weights and the data.

TLDR: “What neuron type should I use?” Use the ReLU non-linearity, be careful with your learning rates and possibly monitor the fraction of “dead” units in a network.

For regular neural networks, the most common layer type is the fully-connected layer in which neurons between two adjacent layers are fully pairwise connected, but neurons within a single layer share no connections.

Working with the two example networks in the above picture: To give you some context, modern Convolutional Networks contain on orders of 100 million parameters and are usually made up of approximately 10-20 layers (hence deep learning).

The full forward pass of this 3-layer neural network is then simply three matrix multiplications, interwoven with the application of the activation function: In the above code, W1,W2,W3,b1,b2,b3 are the learnable parameters of the network.

Notice also that instead of having a single input column vector, the variable x could hold an entire batch of training data (where each input example would be a column of x) and then all examples would be efficiently evaluated in parallel.

Neural Networks work well in practice because they compactly express nice, smooth functions that fit well with the statistical properties of data we encounter in practice, and are also easy to learn using our optimization algorithms (e.g.

Similarly, the fact that deeper networks (with multiple hidden layers) can work better than a single-hidden-layer networks is an empirical observation, despite the fact that their representational power is equal.

As an aside, in practice it is often the case that 3-layer neural networks will outperform 2-layer nets, but going even deeper (4,5,6-layer) rarely helps much more.

We could train three separate neural networks, each with one hidden layer of some size and obtain the following classifiers: In the diagram above, we can see that Neural Networks with more neurons can express more complicated functions.

For example, the model with 20 hidden neurons fits all the training data but at the cost of segmenting the space into many disjoint red and green decision regions.

The subtle reason behind this is that smaller networks are harder to train with local methods such as Gradient Descent: It’s clear that their loss functions have relatively few local minima, but it turns out that many of these minima are easier to converge to, and that they are bad (i.e.

Conversely, bigger neural networks contain significantly more local minima, but these minima turn out to be much better in terms of their actual loss.

In practice, what you find is that if you train a small network the final loss can display a good amount of variance - in some cases you get lucky and converge to a good place but in some cases you get trapped in one of the bad minima.

Binarized Neural Networks - NIPS 2016

Work by: Itay Hubara*, Matthieu Courbariaux*, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio This paper introduces deep neural networks with weights and activation constrained to +1 and -1.

NIMH Non-Invasive Brain Stimulation E-Field Modeling Workshop

The National Institute of Mental Health (NIMH/NIH) hosted a Non-Invasive Brain Stimulation E-Field Modeling Workshop on Saturday, November 11, 2017, immediately prior to the Society for Neuroscienc...

Geospatial Forum: Dr. Brian Reich

Speaker: Dr. Brian Reich | Associate Professor | Department of Statistics | NC State University Abstract: Forensic analyses are often concerned with identifying the spatial source of biological...