Generalizing Outside the Training Set: When Can Neural Networks Learn Identity Effects?

Often in language and other areas of cognition, whether two components of an object are identical or not determine whether it is well formed. We call such constraints identity effects. When developing a system to learn well-formedness from examples, it is easy enough to build in an identify effect. But can identity effects be learned from the data without explicit guidance? We provide a simple framework in which we can rigorously prove that algorithms satisfying simple criteria cannot make the correct inference. We then show that a broad class of algorithms including deep neural networks with standard architecture and training with backpropagation satisfy our criteria, dependent on the encoding of inputs. Finally, we demonstrate our theory with computational experiments in which we explore the effect of different input encodings on the ability of algorithms to generalize to novel inputs.

Stability and Fluctuations in a Simple Model of Phonetic Category Change

In spoken languages, speakers divide up the space of phonetic possibilities into different regions, corresponding to different phonemes. We consider a simple exemplar model of how this division of phonetic space varies over time among a population of language users. In the particular model we consider, we show that, once the system is initialized with a given set of phonemes, that phonemes do not become extinct: all phonemes will be maintained in the system for all time. This is in contrast to what is observed in more complex models. Furthermore, we show that the boundaries between phonemes fluctuate and we quantitatively study the fluctuations in a simple instance of our model. These results prepare the ground for more sophisticated models in which some phonemes go extinct or new phonemes emerge through other processes.

Discrete symbolic optimization and Boltzmann sampling by continuous neural dynamics: Gradient Symbolic Computation

Gradient Symbolic Computation is proposed as a means of solving discrete global optimization problems using a neurally plausible continuous stochastic dynamical system. Gradient symbolic dynamics involves two free parameters that must be adjusted as a function of time to obtain the global maximizer at the end of the computation. We provide a summary of what is known about the GSC dynamics for special cases of settings of the parameters, and also establish that there is a schedule for the two parameters for which convergence to the correct answer occurs with high probability. These results put the empirical results already obtained for GSC on a sound theoretical footing.

Which Learning Algorithms Can Generalize Identity-Based Rules to Novel Inputs?

We propose a novel framework for the analysis of learning algorithms that allows us to say when such algorithms can and cannot generalize certain patterns from training data to test data. In particular we focus on situations where the rule that must be learned concerns two components of a stimulus being identical. We call such a basis for discrimination an identity-based rule. Identity-based rules have proven to be difficult or impossible for certain types of learning algorithms to acquire from limited datasets. This is in contrast to human behaviour on similar tasks. Here we provide a framework for rigorously establishing which learning algorithms will fail at generalizing identity-based rules to novel stimuli. We use this framework to show that such algorithms are unable to generalize identity-based rules to novel inputs unless trained on virtually all possible inputs. We demonstrate these results computationally with a multilayer feedforward neural network.