AI News, NIPS Proceedingsβ

NIPS Proceedingsβ

Part of: Advances in Neural Information Processing Systems 27 (NIPS 2014) Many machine learning approaches are characterized by information constraints on how they interact with the training data.

For example, are there learning problems where any algorithm which has small memory footprint (or can use any bounded number of bits from each example, or has certain communication constraints) will perform worse than what is possible without such constraints?

Working memory

Working memory is a cognitive system with a limited capacity that is responsible for temporarily holding information available for processing.[1] Working memory is important for reasoning and the guidance of decision-making and behavior.[2][3] Working memory is often used synonymously with short-term memory, but some theorists consider the two forms of memory distinct, assuming that working memory allows for the manipulation of stored information, whereas short-term memory only refers to the short-term storage of information.[2][4] Working memory is a theoretical concept central to cognitive psychology, neuropsychology, and neuroscience.

The theory proposed a model containing three components: the central executive, the phonological loop, and the visuospatial sketchpad with the central executive functioning as a control center of sorts, directing info between the phonological and visuospatial components.[12] The central executive is responsible inter alia for directing attention to relevant information, suppressing irrelevant information and inappropriate actions, and coordinating cognitive processes when more than one task is simultaneously performed.

In general, memory span for verbal contents (digits, letters, words, etc.) depends on the phonological complexity of the content (i.e., the number of phonemes, the number of syllables),[21] and on the lexical status of the contents (whether the contents are words known to the person or not).[22] Several other factors affect a person's measured span, and therefore it is difficult to pin down the capacity of short-term or working memory to a number of chunks.

Other tasks that do not have this dual-task nature have also been shown to be good measures of working memory capacity.[26] Whereas Daneman and Carpenter believed that the combination of 'storage' (maintenance) and processing is needed to measure working memory capacity, we know now that the capacity of working memory can be measured with short-term memory tasks that have no additional processing component.[27][28] Conversely, working memory capacity can also be measured with certain processing tasks that don't involve maintenance of information.[29][30] The question of what features a task must have to qualify as a good measure of working memory capacity is a topic of ongoing research.

One is that a limited pool of cognitive resources needed to keep representations active and thereby available for processing, and for carrying out processes.[35] Another hypothesis is that memory traces in working memory decay within a few seconds, unless refreshed through rehearsal, and because the speed of rehearsal is limited, we can maintain only a limited amount of information.[36] Yet another idea is that representations held in working memory interfere with each other.[37] The assumption that the contents of short-term or working memory decay over time, unless decay is prevented by rehearsal, goes back to the early days of experimental research on short-term memory.[38][39] It is also an important assumption in the multi-component theory of working memory.[40] The most elaborate decay-based theory of working memory to date is the 'time-based resource sharing model'.[41] This theory assumes that representations in working memory decay unless they are refreshed.

In a series of experiments, Barrouillet and colleagues have shown that memory for lists of letters depends neither on the number of processing steps nor the total time of processing but on cognitive load.[42] Resource theories assume that the capacity of working memory is a limited resource that must be shared between all representations that need to be maintained in working memory simultaneously.[43] Some resource theorists also assume that maintenance and concurrent processing share the same resource;[35] this can explain why maintenance is typically impaired by a concurrent processing demand.

For example, remembering numbers while processing spatial information, or remembering spatial information while processing numbers, impair each other much less than when material of the same kind must be remembered and processed.[48] Also, remembering words and processing digits, or remembering digits and processing words, is easier than remembering and processing materials of the same category.[49] These findings are also difficult to explain for the decay hypothesis, because decay of memory representations should depend only on how long the processing task delays rehearsal or recall, not on the content of the processing task.

The capacity of working memory increases gradually over childhood[52] and declines gradually in old age.[53] Measures of performance on tests of working memory increase continuously between early childhood and adolescence, while the structure of correlations between different tests remains largely constant.[52] Starting with work in the Neo-Piagetian tradition,[54][55] theorists have argued that the growth of working-memory capacity is a major driving force of cognitive development.

This hypothesis has received substantial empirical support from studies showing that the capacity of working memory is a strong predictor of cognitive abilities in childhood.[56] Particularly strong evidence for a role of working memory for development comes from a longitudinal study showing that working-memory capacity at one age predicts reasoning ability at a later age.[57] Studies in the Neo-Piagetian tradition have added to this picture by analyzing the complexity of cognitive tasks in terms of the number of items or relations that have to be considered simultaneously for a solution.

A meta-analytic summary of research with Klingberg's training program up to 2011 shows that this training has at best a negligible effect on tests of intelligence and of attention[69] In another influential study, training with a working memory task (the dual n-back task) has improved performance on a fluid intelligence test in healthy young adults.[70] The improvement of fluid intelligence by training with the n-back task was replicated in 2010,[71] but two studies published in 2012 failed to reproduce the effect.[72][73] The combined evidence from about 30 experimental studies on the effectiveness of working-memory training has been evaluated by several meta-analyses.[74][75] The authors of these meta-analyses disagree in their conclusions as to whether or not working-memory training improves intelligence.

Later research has shown similar delay-active neurons also in the posterior parietal cortex, the thalamus, the caudate, and the globus pallidus.[78] The work of Goldman-Rakic and others showed that principal sulcal, dorsolateral PFC interconnects with all of these brain regions, and that neuronal microcircuits within PFC are able to maintain information in working memory through recurrent excitatory glutamate networks of pyramidal cells that continue to fire throughout the delay period.[79] These circuits are tuned by lateral inhibition from GABAergic interneurons.[80] The neuromodulatory arousal systems markedly alter PFC working memory function;

The activation during verbal working memory tasks can be broken down into one component reflecting maintenance, in the left posterior parietal cortex, and a component reflecting subvocal rehearsal, in the left frontal cortex (Broca's area, known to be involved in speech production).[90] There is an emerging consensus that most working memory tasks recruit a network of PFC and parietal areas.

Right Brodmann 10 and 47 in the ventral frontal cortex were involved more frequently with demand for manipulation such as dual-task requirements or mental operations, and Brodmann 7 in the posterior parietal cortex was also involved in all types of executive function.[96] Working memory has been suggested to involve two processes with different neuroanatomical locations in the frontal and parietal lobes.[97] First, a selection operation that retrieves the most relevant item, and second an updating operation that changes the focus of attention made upon it.

Updating the attentional focus has been found to involve the transient activation in the caudal superior frontal sulcus and posterior parietal cortex, while increasing demands on selection selectively changes activation in the rostral superior frontal sulcus and posterior cingulate/precuneus.[97] Articulating the differential function of brain regions involved in working memory is dependent on tasks able to distinguish these functions.[98] Most brain imaging studies of working memory have used recognition tasks such as delayed recognition of one or several stimuli, or the n-back task, in which each new stimulus in a long series must be compared to the one presented n steps back in the series.

This phenomenon was first discovered in animal studies by Arnsten and colleagues,[101] who have shown that stress-induced catecholamine release in PFC rapidly decreases PFC neuronal firing and impairs working memory performance through feedforward, intracellular signaling pathways.[102] Exposure to chronic stress leads to more profound working memory deficits and additional architectural changes in PFC, including dendritic atrophy and spine loss,[103] which can be prevented by inhibition of protein kinase C signaling.[104] fMRI research has extended this research to humans, and confirms that reduced working memory caused by acute stress links to reduced activation of the PFC, and stress increased levels of catecholamines.[105] Imaging studies of medical students undergoing stressful exams have also shown weakened PFC functional connectivity, consistent with the animal studies.[106] The marked effects of stress on PFC structure and function may help to explain how stress can cause or exacerbate mental illness.

Adolescents who start drinking at a young age show a decreased BOLD response in these brain regions.[110] Alcohol dependent young women in particular exhibit less of a BOLD response in parietal and frontal cortices when performing a spatial working memory task.[111] Binge drinking, specifically, can also affect one's performance on working memory tasks, particularly visual working memory.[112][113] Additionally, there seems to be a gender difference in regards to how alcohol affects working memory.

Initial evidence for this relation comes from the correlation between working-memory capacity and reading comprehension, as first observed by Daneman and Carpenter (1980)[121] and confirmed in a later meta-analytic review of several studies.[122] Subsequent work found that working memory performance in primary school children accurately predicted performance in mathematical problem solving.[123] One longitudinal study showed that a child's working memory at 5 years old is a better predictor of academic success than IQ.[124] In a large-scale screening study, one in ten children in mainstream classrooms were identified with working memory deficits.

In children with learning disabilities such as dyslexia, ADHD, and developmental coordination disorder, a similar pattern is evident.[128][129][130][131] There is some evidence that optimal working memory performance links to the neural ability to focus attention on task-relevant information and to ignore distractions,[132] and that practice-related improvement in working memory is due to increasing these abilities.[133] One line of research suggests a link between the working memory capacities of a person and their ability to control the orientation of attention to stimuli in the environment.[134] Such control enables people to attend to information important for their current goals, and to ignore goal-irrelevant stimuli that tend to capture their attention due to their sensory saliency (such as an ambulance siren).

The direction of attention according to one's goals is assumed to rely on 'top-down' signals from the pre-frontal cortex (PFC) that biases processing in posterior cortical areas.[135] Capture of attention by salient stimuli is assumed to be driven by 'bottom-up' signals from subcortical structures and the primary sensory cortices.[136] The ability to override 'bottom-up' capture of attention differs between individuals, and this difference has been found to correlate with their performance in a working-memory test for visual information.[134] Another study, however, found no correlation between the ability to override attentional capture and measures of more general working-memory capacity.[137] An impairment of working memory functioning is normally seen in several neural disorders: ADHD: Several authors[138] have proposed that symptoms of ADHD arise from a primary deficit in a specific executive function (EF) domain such as working memory, response inhibition or a more general weakness in executive control.[139] A meta-analytical review cites several studies that found significant lower group results for ADHD in spatial and verbal working memory tasks, and in several other EF tasks.

Lecture 15 | Efficient Methods and Hardware for Deep Learning

In Lecture 15, guest lecturer Song Han discusses algorithms and specialized hardware that can be used to accelerate training and inference of deep learning ...

Predictive Learning, NIPS 2016 | Yann LeCun, Facebook Research

Deep learning has been at the root of significant progress in many application areas, such as computer perception and natural language processing. But almost ...

Thomas Serre: Towards a system-level theory of computation in the visual cortex

Perception involves a complex interaction between feedforward (bottom-up) sensory-driven inputs and feedback (top-down) attention and memory-driven ...

NIPS 2011 Big Learning - Algorithms, Systems, & Tools Workshop: Hazy - Making Data-driven...

Big Learning Workshop: Algorithms, Systems, and Tools for Learning at Scale at NIPS 2011 Invited Talk: Hazy: Making Data-driven Statistical Applications ...

Lecture 12 | Visualizing and Understanding

In Lecture 12 we discuss methods for visualizing and understanding the internal mechanisms of convolutional networks. We also discuss the use of ...

6/7/14 Circuits for Intelligence - Matt Wilson: Hippocampal Memory Reactivation

Halving Global Cardiovascular Disease Mortality

Cardiovascular disease (CVD), which includes heart attack, stroke and heart failure, is the leading cause of death in the world today, with an estimated 17.5 ...

Seth Lloyd: Quantum Machine Learning

Seth Lloyd visited the Quantum AI Lab at Google LA to give a tech talk on "Quantum Machine Learning." This talk took place on January 29, 2014. Speaker Info: ...

Lecture 7: Introduction to TensorFlow

Lecture 7 covers Tensorflow. TensorFlow is an open source software library for numerical computation using data flow graphs. It was originally developed by ...

Computational Analysis Methods and Issues in Human Cognitive Neuroscience

Google Tech Talk January 14, 2010 ABSTRACT Presented by Bradley Voytek. There is a massive, relatively uncoordinated effort underway to map out the ...