# AI News, Neural Networks with R – A Simple Example

- On Tuesday, March 6, 2018
- By Read More

## Neural Networks with R – A Simple Example

In this tutorial a neural network (or Multilayer perceptron depending on naming convention) will be build that is able to take a number and calculate the square root (or as close to as possible).

There is lots of good literature on neural networks freely available on the internet, a good starting point is the neural network handout by Dr Mark Gayles at the Engineering Department Cambridge University http://mi.eng.cam.ac.uk/~mjfg/local/I10/i10_hand4.pdf, it covers just enough to get an understanding of what a neural network is and what it can do without being too mathematically advanced to overwhelm the reader.

- On Saturday, March 10, 2018
- By Read More

## Regression Tutorial with the Keras Deep Learning Library in Python

Keras is a deep learning library that wraps the efficient numerical libraries Theano and TensorFlow.

The dataset describes 13 numerical properties of houses in Boston suburbs and is concerned with modeling the price of houses in those suburbs in thousands of dollars.

Reasonable performance for models evaluated using Mean Squared Error (MSE) are around 20 in squared thousands of dollars (or $4,500 if you take the square root).

This is desirable, because scikit-learn excels at evaluating models and will allow us to use powerful data preparation and model evaluation schemes with very few lines of code.

It is a simple model that has a single fully connected hidden layer with the same number of neurons as input attributes (13).

No activation function is used for the output layer because it is a regression problem and we are interested in predicting numerical values directly without transform.

It is a desirable metric because by taking the square root gives us an error value we can directly understand in the context of the problem (thousands of dollars).

We create an instance and pass it both the name of the function to create the neural network model as well as some parameters to pass along to the fit() function of the model later, such as the number of epochs and batch size.

The result reports the mean squared error including the average and standard deviation (average variance) across all 10 folds of the cross validation evaluation.

further extension of this section would be to similarly apply a rescaling to the output variable such as normalizing it to the range of 0-1 and use a Sigmoid or similar activation function on the output layer to narrow output predictions to the same range.

Our network topology now looks like: We can evaluate this network topology in the same way as above, whilst also using the standardization of the dataset that above was shown to improve performance.

In this section we evaluate the effect of keeping a shallow network architecture and nearly doubling the number of neurons in the one hidden layer.

Our network topology now looks like: We can evaluate the wider network topology using the same scheme as above: Building the model does see a further drop in error to about 21 thousand squared dollars.

Through this tutorial you learned how to develop and evaluate neural network models, including: Do you have any questions about the Keras deep learning library or about this post?

- On Saturday, March 10, 2018
- By Read More

## Creating & Visualizing Neural Network in R

Neural network is an information-processing machine and can be viewed as analogous to human nervous system.

Just like human nervous system, which is made up of interconnected neurons, a neural network is made up of interconnected information processing units.

In fact, neural network draws its strength from parallel processing of information, which allows it to deal with non-linearity.

Neural network becomes handy to infer meaning and detect patterns from complex data sets.

users view the input and output of a neural network but remain clueless about the knowledge generating process.

We hope that the article will help readers learn about the internal mechanism of a neural network and get hands-on experience to implement it in R.

neural network is a model characterized by an activation function, which is used by interconnected information processing units to transform input into output.

The first layer of the neural network receives the raw input, processes it and passes the processed information to the hidden layers.

trains itself from the data, which has a known outcome and optimizes its weights for a better prediction in situations with unknown outcome.

single layer neural network, is the most basic form of a neural network. A perceptron receives multidimensional input and processes it using a weighted summation and an activation function.

This error is backpropagated to all the units such that the error at each unit is proportional to the contribution of that unit towards total error at the output unit.

Please set working directory in R using setwd( ) function, and keep cereal.csv in the working directory.

Training set is used to find the relationship between dependent and independent variables while the test set assesses the performance of the model.

The major problem of residual evaluation methods is that it does not inform us about the behaviour of our model when new data is introduced.

We tried to deal with the “new data” problem by splitting our data into training and test set, constructing the model on training set and evaluating the model by calculating RMSE for the test set.

A limitation of the holdout method is the variance of performance evaluation metric, in our case RMSE, can be high based on the elements assigned to training and test set.

The complete data is partitioned into k equal subsets and each time a subset is assigned as test set while others are used for training the model.

Every data point gets a chance to be in test set and training set, thus this method reduces the dependence of performance on test-training split and reduces the variance of performance metrics.

The extreme case of k-fold cross validation will occur when k is equal to number of data points.

It would mean that the predictive model is trained over all the data points except one data point, which takes the role of a test set.

The number of elements in the training set, j, are varied from 10 to 65 and for each j, 100 samples are drawn form the dataset.

4 shows that the median RMSE across 100 samples when length of training set is fixed to 65 is 5.70.

We show that model accuracy increases when training set is large. Before using the model for prediction, it is important to check the robustness of performance through cross validation.

We have provided commented R code throughout the article to help readers with hands on experience of using neural networks.

- On Saturday, March 10, 2018
- By Read More

It involves subtracting the mean across every individual feature in the data, and has the geometric interpretation of centering the cloud of data around the origin along every dimension.

It only makes sense to apply this preprocessing if you have a reason to believe that different input features have different scales (or units), but they should be of approximately equal importance to the learning algorithm. In

case of images, the relative scales of pixels are already approximately equal (and in range from 0 to 255), so it is not strictly necessary to perform this additional preprocessing step.

Then, we can compute the covariance matrix that tells us about the correlation structure in the data: The (i,j) element of the data covariance matrix contains the covariance between i-th and j-th dimension of the data.

To decorrelate the data, we project the original (but zero-centered) data into the eigenbasis: Notice that the columns of U are a set of orthonormal vectors (norm of 1, and orthogonal to each other), so they can be regarded as basis vectors.

This is also sometimes refereed to as Principal Component Analysis (PCA) dimensionality reduction: After this operation, we would have reduced the original dataset of size [N x D] to one of size [N x 100], keeping the 100 dimensions of the data that contain the most variance.

The geometric interpretation of this transformation is that if the input data is a multivariable gaussian, then the whitened data will be a gaussian with zero mean and identity covariance matrix.

One weakness of this transformation is that it can greatly exaggerate the noise in the data, since it stretches all dimensions (including the irrelevant dimensions of tiny variance that are mostly noise) to be of equal size in the input.

Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative.

The idea is that the neurons are all random and unique in the beginning, so they will compute distinct updates and integrate themselves as diverse parts of the full network.

The implementation for one weight matrix might look like W = 0.01* np.random.randn(D,H), where randn samples from a zero mean, unit standard deviation gaussian.

With this formulation, every neuron’s weight vector is initialized as a random vector sampled from a multi-dimensional gaussian, so the neurons point in random direction in the input space.

That is, the recommended heuristic is to initialize each neuron’s weight vector as: w = np.random.randn(n) / sqrt(n), where n is the number of its inputs.

The sketch of the derivation is as follows: Consider the inner product \(s = \sum_i^n w_i x_i\) between the weights \(w\) and input \(x\), which gives the raw activation of a neuron before the non-linearity.

And since \(\text{Var}(aX) = a^2\text{Var}(X)\) for a random variable \(X\) and a scalar \(a\), this implies that we should draw from unit gaussian and then scale it by \(a = \sqrt{1/n}\), to make its variance \(1/n\).

In this paper, the authors end up recommending an initialization of the form \( \text{Var}(w) = 2/(n_{in} + n_{out}) \) where \(n_{in}, n_{out}\) are the number of units in the previous layer and the next layer.

A more recent paper on this topic, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification by He et al., derives an initialization specifically for ReLU neurons, reaching the conclusion that the variance of neurons in the network should be \(2.0/n\).

This gives the initialization w = np.random.randn(n) * sqrt(2.0/n), and is the current recommendation for use in practice in the specific case of neural networks with ReLU neurons.

Another way to address the uncalibrated variances problem is to set all weight matrices to zero, but to break symmetry every neuron is randomly connected (with weights sampled from a small gaussian as above) to a fixed number of neurons below it.

For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient.

However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization.

A recently developed technique by Ioffe and Szegedy called Batch Normalization alleviates a lot of headaches with properly initializing neural networks by explicitly forcing the activations throughout a network to take on a unit gaussian distribution at the beginning of the training.

In the implementation, applying this technique usually amounts to insert the BatchNorm layer immediately after fully connected layers (or convolutional layers, as we’ll soon see), and before non-linearities.

It is common to see the factor of \(\frac{1}{2}\) in front because then the gradient of this term with respect to the parameter \(w\) is simply \(\lambda w\) instead of \(2 \lambda w\).

Lastly, notice that during gradient descent parameter update, using the L2 regularization ultimately means that every weight is decayed linearly: W += -lambda * W towards zero.

L1 regularization is another relatively common form of regularization, where for each weight \(w\) we add the term \(\lambda \mid w \mid\) to the objective.

Another form of regularization is to enforce an absolute upper bound on the magnitude of the weight vector for every neuron and use projected gradient descent to enforce the constraint.

In practice, this corresponds to performing the parameter update as normal, and then enforcing the constraint by clamping the weight vector \(\vec{w}\) of every neuron to satisfy \(\Vert \vec{w} \Vert_2 <

Vanilla dropout in an example 3-layer Neural Network would be implemented as follows: In the code above, inside the train_step function we have performed dropout twice: on the first hidden layer and on the second hidden layer.

It can also be shown that performing this attenuation at test time can be related to the process of iterating over all the possible binary masks (and therefore all the exponentially many sub-networks) and computing their ensemble prediction.

Since test-time performance is so critical, it is always preferable to use inverted dropout, which performs the scaling at train time, leaving the forward pass at test time untouched.

Inverted dropout looks as follows: There has a been a large amount of research after the first introduction of dropout that tries to understand the source of its power in practice, and its relation to the other regularization techniques.

As we already mentioned in the Linear Classification section, it is not common to regularize the bias parameters because they do not interact with the data through multiplicative interactions, and therefore do not have the interpretation of controlling the influence of a data dimension on the final objective.

For example, a binary classifier for each category independently would take the form: where the sum is over all categories \(j\), and \(y_{ij}\) is either +1 or -1 depending on whether the i-th example is labeled with the j-th attribute, and the score vector \(f_j\) will be positive when the class is predicted to be present and negative otherwise.

A binary logistic regression classifier has only two classes (0,1), and calculates the probability of class 1 as: Since the probabilities of class 1 and 0 sum to one, the probability for class 0 is \(P(y = 0 \mid x;

The expression above can look scary but the gradient on \(f\) is in fact extremely simple and intuitive: \(\partial{L_i} / \partial{f_j} = y_{ij} - \sigma(f_j)\) (as you can double check yourself by taking the derivatives).

The L2 norm squared would compute the loss for a single example of the form: The reason the L2 norm is squared in the objective is that the gradient becomes much simpler, without changing the optimal parameters since squaring is a monotonic operation.

For example, if you are predicting star rating for a product, it might work much better to use 5 independent classifiers for ratings of 1-5 stars instead of a regression loss.

If you’re certain that classification is not appropriate, use the L2 but be careful: For example, the L2 is more fragile and applying dropout in the network (especially in the layer right before the L2 loss) is not a great idea.

- On Saturday, March 10, 2018
- By Read More

## Neural Network Tutorial

Starting with measured data from some known or unknown source, a neural network may be trained to perform classification, estimation, simulation, and prediction of the underlying process generating the data.

The Neural Networks package supports several function estimation techniques that may be described in terms of different types of neural networks and associated The general area of artificial neural networks has its roots in our understanding of the human brain.

Efforts that followed gave rise to various models of biological neural network structures and learning algorithms.

This is in contrast to the computational models found in this package, which are only concerned with artificial neural networks as a tool for solving different types of problems where unknown relationships are sought among given data.

Still, much of the nomenclature in the neural network arena has its origins in biological neural networks, and thus, the original terminology will be used alongside with more traditional nomenclature from statistics and engineering.

Let the input to a neural network be denoted by x, a real-valued (row) vector of arbitrary dimensionality or length.

Let the network output be denoted by , an approximation of the desired output y, also a real-valued vector having one or more components, and the number of outputs from the network.

Generally, a neural network is a structure involving weighted interconnections among neurons, or units, which are most often nonlinear scalar transformations, but which can also be linear.

Figure 1 shows an example of a one-hidden-layer neural network with three inputs, x = {, , } that, along with a unity bias input, feed each of the two neurons comprising the hidden layer.

The two outputs from this layer and a unity bias are then fed into the single output layer neuron, yielding the scalar output, .

Generally, a neuron is structured to process multiple inputs, including the unity bias, in a nonlinear way, producing a single output.

By inspection of the Figure 1, the output of the network is given by involving the various parameters of the network, the weights {,,,}.

When algorithmic aspects, independent of the exact structure of the neural network, are discussed, then this compact form becomes more convenient to use than an explicit one, such as that of Eq.

Probably the simplest way, and often the best, is to test the neural network on a data set that was not used for training, but which was generated under similar conditions.

However, depending on the origin of the data, and the intended use of the obtained neural network model, the function approximation problem may be subdivided into several types of problems.

Different types of function approximation problems are described in Features of This Package includes a table giving an overview of the supported neural networks and the particular types of problems they are intended to address.

When input data originates from a function with real-valued outputs over a continuous range, the neural network is said to perform a traditional function approximation.

A more advanced model of the second example might use gender as a second input in order to derive a more accurate estimate of the shoe size.

Two examples of dynamic system problems are: (1) predicting the price of a state bond, or that of some other financial instrument;

The Neural Network package supports both linear and nonlinear models and methods in the form of neural network structures and associated learning algorithms.

Dynamic neural networks can be either feedforward in structure or employ radial basis functions, and they must accommodate memory for past information.

In the context of neural networks, classification involves deriving a function that will separate data into categories, or classes, characterized by a distinct set of features.

This function is mechanized by a so-called network classifier, which is trained using data from the different classes as inputs, and vectors indicating the true class as outputs.

network classifier typically maps a given input vector to one of a number of classes represented by an equal number of outputs, by producing 1 at the output class and 0 elsewhere.

As such, the neural network cannot be trained to produce a desired output in a supervised way, but must instead look for hidden structures in the input data without supervision, employing so-called self-organizing.

Structures in data manifest themselves as constellations of clusters that imply levels of correlation among the raw data and a consequent reduction in dimensionality and increased information in coding efficiency.

Specifically, a particular input data vector that falls within a given cluster could be represented by its unique centroid within some squared error.

Such networks, known as self-organizing maps or Kohonen networks, may be interpreted loosely as being nonlinear projections of the original data onto a one- or two-dimensional space.

One way to check for quality is to view graphical representations of the data in question, in the hope of selecting a reasonable subset while eliminating problematic parts.

For this purpose, you can use any suitable Mathematica plotting function or employ other such functions that come with the Neural Networks package especially designed to visualize the data in classification, time series, and dynamical system problems.

Mathematically, the linear model gives rise to the following simple equation for the output Linear models are called regression models in traditional statistics.

Figure 4.

As indicated, the weighted sum of the inputs and the unity bias are first summed and then processed by a step function to yield the output where {, ..., } are the weights applied to the input vector and b is the bias weight.

In two-dimensional problems (where x is a two-component row vector), the classes may be separated by a straight line, and in higher-dimensional problems it means that the classes are separable by a hyperplane.

Also, important insights may be gained from using the perceptron, which may shed some light in considering more complicated neural network models.

More specifically, as individual inputs are presented to the perceptron, its weights are adjusted iteratively by the training algorithm so as to produce the correct class mapping at the output.

This training process continues until the perceptron correctly classifies all the training data or when a maximum number of iterations has been reached.

Classification problems involving a number of classes greater than two can be handled by a multi-output perceptron that is defined as a number of perceptrons in parallel.

The training process of such a multi-output perceptron structure attempts to map each input of the training data to the correct class by iteratively adjusting the weights to produce 1 at the output of the corresponding perceptron and 0 at the outputs of all the remaining outputs.

It may also be that the perceptron classifier cannot make a decision for a subset of input vectors because of the nature of the data or insufficient complexity of the network structure itself.

perceptron is defined parametrically by its weights {w,b}, where w is a column vector of length equal to the dimension of the input vector x and b is a scalar.

}, the output of a perceptron is described in compact form by This description can be used also when a set of input vectors is considered.

The weights {w,b} are obtained by iteratively training the perceptron with a known data set containing input-output pairs, one input vector in each row of a matrix x, and one output in each row of a matrix y, as described in Data Format.

Given N such pairs in the data set, the training algorithm is defined by where i is the iteration number, is a scalar step size, and =y-

(x,,) is a column vector with N-components of classification errors corresponding to the N data samples of the training set.

At any iteration i, values of 0 indicate that the corresponding data samples have been classified correctly, while all the others have been classified incorrectly.

(6) begins with initial values for the weights {w,b} and i=0, and iteratively updates these weights until all data samples have been classified correctly or the iteration number has reached a maximum value, .

The step size η, or learning rate as it is often called, has the following default value By compensating for the range of the input data, x, and for the number of data samples, N, this default value of η

However, although larger values of might accelerate the training process, they may induce oscillations that may slow down the convergence.

Each neuron performs a weighted summation of the inputs, which then passes a nonlinear activation function σ, also called the neuron function.

In general, the neural network model will be represented by the compact notation g(θ,x) whenever the exact structure of the neural network is not necessary in the context of a discussion.

Note that the size of the input and output layers are defined by the number of inputs and outputs of the network and, therefore, only the number of hidden neurons has to be specified when the network is defined.

The package supports FF neural networks with any number of hidden layers and any number of neurons (hidden neurons) in each layer.

This is, of course, not a very useful rule, and in practice you have to experiment with different designs and compare the results, to find the most suitable neural network model for the problem at hand.

Figure 7.

Mathematically, the RBF network, including a linear part, produces an output given by where nb is the number of neurons, each containing a basis function.

The parameters of the RBF network consist of the positions of the basis functions , the inverse of the width of the basis functions , the weights in output sum , and the parameters of the linear part ,...,.

Then you can use the generic description g(θ,x) of the neural network model, where g is the network function and x is the input to the network.In training, the network the parameters are tuned so that the training data fits the network model Eq.

Problems where these networks are useful include: Suppose you have chosen an FF or RBF network and you have already decided on the exact structure, the number of layers, and the number of neurons in the different layers.

that is, Often it is more convenient to use the root-mean-square error (RMSE) when evaluating the quality of a model during and after training, since it can be compared with the output signal directly.

The training is terminated prior to the specified number of iteration if any of the following conditions are satisfied: The Gauss-Newton method is a fast and reliable algorithm that may be used for a large variety of minimization problems.

Then, the algorithm continues with a new iteration.The training terminates prior to the specified number of iterations if any of the following conditions are satisfied: The training algorithm in Eq.

The training with the steepest descent method will stop prior to the given number of iterations under the same conditions as the Gauss-Newton method.

system may be modeled by a dynamic neural network that consists of a combination of neural networks of FF or RBF types, and a specification of the input vector to the network.

The input vector, or regressor vector, which it is often called in connection with dynamic systems, contains lagged input and output values of the system specified by three indices: , and .

For example, a problem with three outputs and two inputs ={1,2,1}, ={2,1}, and ={1,0} gives the following regressor: For time-series problems, only has to be chosen.

The neural network part of the dynamic neural network defines a mapping from the regressor space to the output space.

Since the mapping g(θ, ·) is based on neural networks, the dynamic models are called neural ARX and neural AR models, or neural AR(X) as short form for both them.

Figure 10.

Depending on the choice of the mapping g(θ, ·) you obtain a linear or a nonlinear model using an FF network or an RBF network.

The importance of the different Hopfield networks in practical application is limited due to theoretical limitations of the network structure but, in certain situations, they may form interesting models.

}, vectors consisting of +/-1 elements, to be stored in the network, and n is the number of components, the dimension, of the class pattern vectors.

For a discrete-time Hopfield network, the energy of a certain vector x is given by It can be shown that, given an initial state vector x(0), x(t) in Eq.

(27) constitute possible convergence points of the Hopfield network and, ideally, these minima are identical to the class patterns {

The continuous Hopfield network is described by the following differential equation where x(t) is the state vector of the network, W represents the parametric weights, and σ

(25), one can define the energy of a particular state vector x as As for the discrete-time network, it can be shown that given an initial state vector x(0) the state vector x(t) in Eq.

(29) constitute the possible convergence points of the Hopfield network and ideally these minima are identical to the class patterns {

Unsupervised networks, or self-organizing networks, rely only on input data and try to find structures in the input data space.

Another test that can be applied in any number of dimensions is to check for the mean distance between the data points and the obtained cluster centers.

When an unsupervised network is trained, the locations of the codebook vectors are adapted so that the mean Euclidean distance between each data point and its closest codebook vector is minimized.

In this way it is possible to define one- or two-dimensional relations among the codebook vectors, and the obtained SOM unsupervised network becomes a nonlinear mapping from the original data space to the one- or two-dimensional feature space defined by the codebook vectors.

Each class has a subset of the codebook vectors associated to it, and a data vector is classified to be in the class to which the closest codebook vector belongs.

For VQ networks this training algorithm can be used by considering the data and the codebook vectors of a specific class independently of the rest of the data and the rest of the codebook vectors.

In most cases the neural network training is nothing other than minimization and it is, therefore, a good idea to consult standard books on minimization, such as: J.E.