AI News, Scaling deep learning for science

Scaling deep learning for science

Now, researchers are eager to apply this computational technique -- commonly referred to as deep learning -- to some of science's most persistent mysteries.

But because scientific data often looks much different from the data used for animal photos and speech, developing the right artificial neural network can feel like an impossible guessing game for nonexperts.

Using the Titan supercomputer, a research team led by Robert Patton of the US Department of Energy's (DOE's) Oak Ridge National Laboratory (ORNL) has developed an evolutionary algorithm capable of generating custom neural networks that match or exceed the performance of handcrafted artificial intelligence systems.

Better yet, by leveraging the GPU computing power of the Cray XK7 Titan -- the leadership-class machine managed by the Oak Ridge Leadership Computing Facility, a DOE Office of Science User Facility at ORNL -- these auto-generated networks can be produced quickly, in a matter of hours as opposed to the months needed using conventional methods.

Scaled across Titan's 18,688 GPUs, MENNDL can test and train thousands of potential networks for a science problem simultaneously, eliminating poor performers and averaging high performers until an optimal network emerges.

Today's neural networks can consist of thousands or millions of simple computational units -- the 'neurons' -- arranged in stacked layers, like the rows of figures spaced across a foosball table.

During one common form of training, a network is assigned a task (e.g., to find photos with cats) and fed a set of labeled data (e.g., photos of cats and photos without cats).

As the network pushes the data through each successive layer, it makes correlations between visual patterns and predefined labels, assigning values to specific features (e.g., whiskers and paws).

As Titan works through individual networks, new data is fed to the system's nodes asynchronously, meaning once a node completes a task, it's quickly assigned a new task independent of the other nodes' status.

To demonstrate MENNDL's versatility, the team applied the algorithm to several datasets, training networks to identify sub-cellular structures for medical research, classify satellite images with clouds, and categorize high-energy physics data.

Neutrinos, ghost-like particles that pass through your body at a rate of trillions per second, could play a major role in explaining the formation of the early universe and the nature of matter -- if only scientists knew more about them.

The task, known as vertex reconstruction, required a network to analyze images and precisely identify the location where neutrinos interact with the detector -- a challenge for events that produce many particles.

Furthermore, because deep learning requires less mathematical precision than other types of scientific computing, Summit could potentially deliver exascale-level performance for deep learning problems -- the equivalent of a billion billion calculations per second.

In addition to preparing for new hardware, Patton's team continues to develop MENNDL and explore other types of experimental techniques, including neuromorphic computing, another biologically inspired computing concept.

Scaling Deep Learning for Science

Now, researchers are eager to apply this computational technique—commonly referred to as deep learning—to some of science’s most persistent mysteries.

But because scientific data often looks much different from the data used for animal photos and speech, developing the right artificial neural network can feel like an impossible guessing game for nonexperts.

Using the Titan supercomputer, a research team led by Robert Patton of the US Department of Energy’s(DOE’s) Oak Ridge National Laboratory (ORNL) has developed an evolutionary algorithm capable of generating custom neural networks that match or exceed the performance of handcrafted artificial intelligence systems.

Scaled across Titan’s 18,688 GPUs, MENNDL can test and train thousands of potential networks for a science problem simultaneously, eliminating poor performers and averaging high performers until an optimal network emerges.

Instead, the algorithm can quickly do that for them, while they focus on their data and ensuring the problem is well-posed.” Pinning down parameters Inspired by the brain’s web of neurons, deep neural networks are a relatively old concept in neuroscience and computing, first popularized by two University of Chicago researchers in the 1940s.

During one common form of training, a network is assigned a task (e.g., to find photos with cats) and fed a set of labeled data (e.g., photos of cats and photos without cats).

As Titan works through individual networks, new data is fed to the system’s nodes asynchronously, meaning once a node completes a task, it’s quickly assigned a new task independent of the other nodes’ status.

“To really leverage the machine, we set up MENNDL to generate a queue of individual networks to send to the nodes for evaluation as soon as computing power becomes available.” To demonstrate MENNDL’s versatility, the team applied the algorithm to several datasets, training networks to identify sub-cellular structures for medical research, classify satellite images with clouds, and categorize high-energy physics data.

Neutrinos, ghost-like particles that pass through your body at a rate of trillions per second, could play a major role in explaining the formation of the early universe and the nature of matter—if only scientists knew more about them.

The devices capture a large sample of neutrino interactions that can be transformed into basic images through a process called “reconstruction.” Like a slow-motion replay at a sporting event, these reconstructions can help physicists better understand neutrino behavior.

“What Titan does is bring the time to solution down to something practical.” Having recently been awarded another allocation under the Advanced Scientific Computing Research Leadership Computing Challenge program, Perdue’s team is building off its deep learning success by applying MENDDL to additional high-energy physics datasets to generate optimized algorithms.

The reason we’re going through all this work is because we’re getting better performance, and there’s real potential to get more.” AI meets exascale When Titan debuted 5 years ago, its GPU-accelerated architecture boosted traditional modeling and simulation to new levels of detail.

Furthermore, because deep learning requires less mathematical precision than other types of scientific computing, Summit could potentially deliver exascale-level performance for deep learning problems—the equivalent of a billion billion calculations per second.

AI Uses Titan Supercomputer to Create Deep Neural Nets in Less Than a Day

You don’t have to dig too deeply into the archive of dystopian science fiction to uncover the horror that intelligent machines might unleash.

The tech giant announced it was developing automated machine learning (AutoML), writing algorithms that can do some of the heavy lifting by identifying the right neural networks for a specific job.

In another case involving a collaboration with St. Jude Children’s Research Hospital in Memphis, MENNDL improved the error rate of a human-designed algorithm for identifying mitochondria inside 3D electron microscopy images of brain tissue by 30 percent.

“Our software is more [about] pattern detection, let’s say anti-money laundering or fraud detection or which customer is most likely to churn,” Dr. Arno Candel, chief technology officer at H2O.ai, tells Singularity Hub.

“And that kind of insight-generating software is what we call AI here.” The company’s latest product, Driverless AI, promises to deliver the data scientist equivalent of a chessmaster to its customers (the company claims several such grandmasters in its employ and advisory board).

In other words, the system can analyze a raw dataset and, like MENNDL, automatically identify what features should be included in the computer model to make the most of the data based on the best “chess moves” of its grandmasters.

“So we created a virtual data scientist that is relentless at trying these ideas.” Not unlike how the human brain reaches a conclusion, it’s not always possible to understand how a machine, despite being designed by humans, reaches its own solutions.

The lack of transparency is often referred to as the AI “black box.” Experts like Young say we can learn something about the evolutionary process of machine learning by generating millions of neural networks and seeing what works well and what doesn’t.

“It’s not just neural nets that magically come up with some kind of number, but we’re trying to make it statistically significant.” Much digital ink has been spilled over the dearth of skilled data scientists, so automating certain design aspects for developing artificial neural networks makes sense.

“By automating, we are pushing the burden back for the data scientists to actually do something more meaningful, which is think about the problem and see how you can address it differently to make an even bigger impact.” The team at ORNL expects it can also make bigger impacts beginning next year when the lab’s next supercomputer, Summit, comes online.

ORNL researchers use Titan to accelerate design, training of deep learning networks

OAK RIDGE, Tenn., Jan. 10, 2018 – A team of researchers from the Department of Energy’s Oak Ridge National Laboratory has married artificial intelligence and high-performance computing to achieve a peak speed of 20 petaflops in the generation and training of deep learning networks on the laboratory’s Titan supercomputer.

Due to its ability to make sense of massive amounts of data, researchers across the scientific spectrum are eager to refine deep learning and apply it to some of today’s most challenging science problems.

To efficiently design neural networks capable of tackling scientific datasets and expediting breakthroughs, Patton’s team developed two codes for evolving (MENNDL) and fine-tuning (RAvENNA) deep neural network architectures.

“And with a machine like Titan we are able to train an unparalleled number of highly accurate networks.” Titan is a Cray hybrid system, meaning that it uses both traditional CPUs and graphics processing units (GPUs) to tackle complex calculations for big science problems efficiently;

The team’s work demonstrates that with the right high-performance computing system researchers can efficiently train large numbers of networks, which can then be used to help them tackle today’s increasingly data-heavy experiments and simulations.

This efficient design of deep neural networks will enable researchers to deploy highly accurate, custom-designed models, saving both time and money by freeing the scientist from the task of designing a network from the ground up.

The task, known as vertex reconstruction, required a network to analyze images and precisely identify the location where neutrinos interact with one of many targets—a task akin to finding the aerial source of a starburst of fireworks.

To identify the high-performing network, MENNDL evaluated approximately 500,000 neural networks, training them on a data set consisting of 800,000 images of neutrino events, steadily using 18,000 of Titan’s nodes.

Learning goes deep

Deep neural networks—a form of artificial intelligence—have demonstrated mastery of tasks once thought uniquely human.

To expand the benefits of deep learning for science, researchers need new tools to build high-performing neural networks that don’t require specialized knowledge.

Using the Titan supercomputer, a research team led by Robert Patton of the US Department of Energy’s Oak Ridge National Laboratory (ORNL) has developed an algorithm that can generate custom neural networks that match or exceed the performance of handcrafted artificial intelligence systems.

By leveraging the graphics processing unit (GPU) power of the Cray XK7 Titan, these networks can be produced in a matter of hours instead of months when using conventional methods.

Scaled across Titan’s 18,688 GPUs, MENNDL can test and train thousands of potential networks for a science problem simultaneously, eliminating poor performers and averaging high performers until an optimal network emerges.

During one common form of training, a network is assigned a task (e.g., to find photos with cats) and fed a set of labeled data (e.g., photos of cats and photos without cats).

As the network pushes the data through each successive layer, it makes correlations between visual patterns and predefined labels, assigning values to specific features (e.g., whiskers and paws).

As Titan works through individual networks, new data is fed to the system’s nodes asynchronously so that each node is quickly assigned a new task, keeping Titan busy combing through possible configurations.

“To really leverage the machine, we set up MENNDL to generate a queue of individual networks to send to the nodes for evaluation as soon as computing power becomes available.”

To demonstrate MENNDL’s versatility, the team applied the algorithm to several datasets, training networks to identify sub-cellular structures for medical research, classify satellite images with clouds, and categorize high-energy physics data.

The task, known as vertex reconstruction, required a network to analyze images and precisely identify the location where neutrinos interact with the detector—a challenge for events that produce many particles.

An AI for AI: New Algorithm Poised to Fuel Scientific Discovery

Deep learning is performing wonders, automating everything from driving to recognizing speech to composing music.

“MENNDL is about saving people time and making scientific discovery happen faster,” said Steven Young, a research scientist who’s part of ORNL’s Nature Inspired Machine Learning team.

Scaled across Titan’s 18,688 Tesla GPUs, the algorithm simultaneously tests and trains thousands of potential networks to predict those best suited to the job.

In many fields, researchers use existing neural networks or datasets as a launching pad for their deep learning efforts.

That’s not possible for scientists, whose data comes from scientific instruments and looks a lot different from what’s used to teach computers to recognize faces or understand speech.

“At the lab we work with data that’s pulled from a neutrino detector, electron microscope or some other scientific instrument,” Young said.

Neutrinos are subatomic particles that scientists believe could shed light on such mysteries as the origins of the universe and the nature of matter.

Adapting Deep Learning to New Data Using ORNL's Titan Supercomputer

In this video from SC17, Travis Johnston from ORNL presents: Adapting Deep Learning to New Data Using ORNL's Titan Supercomputer. "There has been a ...