# AI News, Deep Learning Basics: Neural Networks, Backpropagation and Stochastic Gradient Descent

- On Wednesday, March 7, 2018
- By Read More

## Deep Learning Basics: Neural Networks, Backpropagation and Stochastic Gradient Descent

Presently, most deep neural networks are trained using GPUs due to the enormous number of parallel computations that they can perform.

However, using GPUs can be prohitive for several reasons DistBelief is a framework for training deep neural networks that avoids GPUs entirely (for the above reasons) and instead performs parallel computing with clusters of commodity machines.

As we shall see later in this post, DistBelief relies heavily on asynchronous message passing which makes the Akka actor framework a suitable alternative.

Each neural unit in the input layer is connected to every neural unit in the hidden layer via a weight (these are illustrated by the arrows in the above diagram).

Training a neural network refers to the process of finding the optimal layer weights that map the inputs to the outputs in a given data set.

Also, let \(\mathbf{x}\) be the output of layer \(l\) (therefore the dimension of \(\mathbf{x}\) is equal to the number of neural units in layer \(l\)).

The forward pass begins at the input layer with the input of the training example acting as the output of the input layer and the above process is repeated for all layers until the output layer is reached at which point we obtain our prediction.

Then working backwards from layer \(k\) to layer \(l\) we compute $$\delta_l = \sigma'(\mathbf{y}_l) \odot (\mathbf{W}^T_{l,k} \cdot \delta_k)$$

Given the gradient \(\nabla_{l,k}\) from the backward pass, we can update the weights \(\mathbf{W}_{l,k}\) via the stochastic graident descent equation $$\mathbf{W}_{l,k}=\mathbf{W}_{l,k} - \eta \cdot \nabla_{l,k}$$

In a multithreaded environment, multiple threads can access and change this state variable at any time, meaning that no thread can be sure what the value of this variable is after it was initially read.

Akka has solved this problem by introducing entities known as actors which encapsulate the mutable state variables and communicate with each other asynchronously via message passing.

Each parameter partition holds the weights for one layer of the model (for example, if the model replicas have 3 layers, then the parameter server has 2 partitions for the weights from layer 1 to layer 2 and the weights from layer 2 to layer 3 respectively).

Therefore, as the model replicas are trained in parallel, they asynchronously read (in the forward pass) and update (in the backward pass) their corresponding weight parameters.

And each replica communicates asynchronously with the central parameter server which is also partitioned across machines (shown by the green grid squares).

When the parameter shard actor is first created it is given a unique shardId, a learning rate for the update step, and a random intial weight.

When a model replica layer requests the latest parameter values, the actor sends them back to the replica layer wrapped in a LatestParameters message.

Once the model actors are created, the data shard waits to receive the ReadyToProcess message at which point a FetchParameters message is sent to each layer actor in the replica which tells them to retrieve the latest version of their weight parameters from their corresponding parameter shards.

The Master actor also creates ParameterShard actors for each layer weight in the model (remember the model is replicated but there is only one set of parameters that each of them read and update).

being able to predict the correct XOR output given an XOR input) is a classic problem in machine learning because the data points are not linearly separable.

In fact, a multilayer perceptron can learn virtually any non-linear function given a sufficient number of neural units in its hidden layer.

This means that our replicas will each have 3 layers with the first two layers having 2 neural units each and an output layer with one neural unit (note that all layers except for the output layer will have an addition bias neural unit as well).

Also, to emphasize data parallelism, we partition our data set into shards of size 2,000 which means that there will be 25 model replicas performing backpropagation in parallel.

Furthermore, in order to make sure that each replica is reading and updating their corresponding parameter shard asynchronously, we log each parameter read and update.

- On Wednesday, March 7, 2018
- By Read More

## Implementing the DistBelief Deep Neural Network Training Framework with Akka

Presently, most deep neural networks are trained using GPUs due to the enormous number of parallel computations that they can perform.

However, using GPUs can be prohitive for several reasons DistBelief is a framework for training deep neural networks that avoids GPUs entirely (for the above reasons) and instead performs parallel computing with clusters of commodity machines.

As we shall see later in this post, DistBelief relies heavily on asynchronous message passing which makes the Akka actor framework a suitable alternative.

Each neural unit in the input layer is connected to every neural unit in the hidden layer via a weight (these are illustrated by the arrows in the above diagram).

Training a neural network refers to the process of finding the optimal layer weights that map the inputs to the outputs in a given data set.

Also, let \(\mathbf{x}\) be the output of layer \(l\) (therefore the dimension of \(\mathbf{x}\) is equal to the number of neural units in layer \(l\)).

The forward pass begins at the input layer with the input of the training example acting as the output of the input layer and the above process is repeated for all layers until the output layer is reached at which point we obtain our prediction.

Then working backwards from layer \(k\) to layer \(l\) we compute $$\delta_l = \sigma'(\mathbf{y}_l) \odot (\mathbf{W}^T_{l,k} \cdot \delta_k)$$

Given the gradient \(\nabla_{l,k}\) from the backward pass, we can update the weights \(\mathbf{W}_{l,k}\) via the stochastic graident descent equation $$\mathbf{W}_{l,k}=\mathbf{W}_{l,k} - \eta \cdot \nabla_{l,k}$$

In a multithreaded environment, multiple threads can access and change this state variable at any time, meaning that no thread can be sure what the value of this variable is after it was initially read.

Akka has solved this problem by introducing entities known as actors which encapsulate the mutable state variables and communicate with each other asynchronously via message passing.

Each parameter partition holds the weights for one layer of the model (for example, if the model replicas have 3 layers, then the parameter server has 2 partitions for the weights from layer 1 to layer 2 and the weights from layer 2 to layer 3 respectively).

Therefore, as the model replicas are trained in parallel, they asynchronously read (in the forward pass) and update (in the backward pass) their corresponding weight parameters.

And each replica communicates asynchronously with the central parameter server which is also partitioned across machines (shown by the green grid squares).

When the parameter shard actor is first created it is given a unique shardId, a learning rate for the update step, and a random intial weight.

When a model replica layer requests the latest parameter values, the actor sends them back to the replica layer wrapped in a LatestParameters message.

Once the model actors are created, the data shard waits to receive the ReadyToProcess message at which point a FetchParameters message is sent to each layer actor in the replica which tells them to retrieve the latest version of their weight parameters from their corresponding parameter shards.

The Master actor also creates ParameterShard actors for each layer weight in the model (remember the model is replicated but there is only one set of parameters that each of them read and update).

being able to predict the correct XOR output given an XOR input) is a classic problem in machine learning because the data points are not linearly separable.

In fact, a multilayer perceptron can learn virtually any non-linear function given a sufficient number of neural units in its hidden layer.

This means that our replicas will each have 3 layers with the first two layers having 2 neural units each and an output layer with one neural unit (note that all layers except for the output layer will have an addition bias neural unit as well).

Also, to emphasize data parallelism, we partition our data set into shards of size 2,000 which means that there will be 25 model replicas performing backpropagation in parallel.

Furthermore, in order to make sure that each replica is reading and updating their corresponding parameter shard asynchronously, we log each parameter read and update.

- On Saturday, February 22, 2020

**Extraordinary Variations of the Mind: Geschwind:Our Brains Berman: Williams SyndromeFisher:Language**

Visit: 1:39 - Our Brains: Life on a Continuum 21:07 - From Genes to Neural Circuits 35:07 - Language at the Extremes The human mind is one of the features that makes..

**Confluence 2017: Keynote by Dr. Vishal Sikka**

Watch Infosys CEO Dr. Vishal Sikka's keynote at Infosys Confluence 2017, 24th May, 2017.

**QANTA vs. Ken Jennings at UW**

Jeopardy! champion Ken Jennings and cutting-edge quiz-playing AI QANTA go head-to-CPU in quiz bowl UW Computer Science and Engineering is pleased to host an event for both trivia aficionados...

**ZEITGEIST MOVING FORWARD sub ITA / ESP / ENG / JAP /spread**

DocumentaryFilms documentary zeitgeist moving forward, italiano espanol 日本語 portugues This film movie is a non-commercial work and is available online for free viewing and no restrictions...

**Zeitgeist: Moving Forward (2011)**

Zeitgeist: Moving Forward (2011) with subtitle. Storyline: A feature length documentary work which presents a case for a needed transition out of the current socioeconomic monetary paradigm...

**ZEITGEIST: MOVING FORWARD | OFFICIAL RELEASE | 2011**

Please support Peter Joseph's new, upcoming film project: "InterReflections" by joining the mailing list and helping: LIKE Peter Joseph @

**Introduction to Dictionary Skills**

A charming introduction to first dictionary skills, to help every child understand how to use dictionaries to find the words they need, and enrich their language.

**Forward 5: JS Live Stream**

**Auburn Coach Wife Kristi Malzahn Agrees with Match & eHarmony: Men are Jerks**

My advice is this: Settle! That's right. Don't worry about passion or intense connection. Don't nix a guy based on his annoying habit of yelling "Bravo!" in movie theaters. Overlook his halitosis...