Magnus Johnsson

What happens in your brain when you imagine something? Well, one very reasonable explanation to this question is provided by the internal simulation hypothesis, proposed by Germund Hesslow, which is related to the mirror neuron theory. The internal simulation hypothesis suggests that when you imagine that you experience something or that you act in a certain way, similar neural activity patterns are developed in the same brain areas as if you actually had these experiences or as if you actually acted in the way you imagined. The neural activity patterns elicited when you, for example, imagine the sight of your home correspond to those elicited had you actually seen your home, and in the same brain areas, which means there are no special areas in the brain for imagining things. Moreover, the imagination of something might in turn elicit new activity patterns in the same or other brain areas. For example, if you imagine that you are standing in front of your home, the corresponding brain activity in perceptual areas of your brain may elicit activity patterns in motor areas of your brain that correspond to carrying out some action, like opening the door. This in turn might again trigger activity patterns in your perceptual areas corresponding to what you would likely see after the opening of that door. Another example is that when you think about eating something for dinner or see a picture of it, you may already feel the taste and smell of it, just by thinking of it or seeing a picture of it. In this way the internal simulation mechanism creates cross-modal expectations, i.e. experiencing, or even imagining, something with one sensory modality, triggers expectations of what would likely accompany it in other sensory modalities. With this mechanism of internal simulation you can imagine long sequences of perceptions and actions and their likely consequences without having to actually carry them out in reality.

The uppermost picture below illustrates what happens in real interaction with the environment. Stimulus S1 causes perceptual activity s1, which causes preparatory response r1 and overt response R1. R1 causes predictable new stimulus S2, which causes perceptual activity s2, which causes preparatory response r2 and overt response R2. The lowermost picture below illustrates what happens in an example of simulated interaction with the environment. Stimulus S1 causes perceptual activity s1, which causes preparatory response r1. Preparatory response r1 elicits, via internal association mechanisms, perceptual activity s2 and the response R1 is inhibited. The perceptual activity s2 in turn elicits motor activity r2 and the response R2 is inhibited.

I have done a lot of work on how to implement such mechanisms in artificial cognitive architectures that can for example control robots, but they could also be useful for more isolated applications like guessing or coming up with likely and reasonable continuations to partial input. For example, software that can come up with and play reasonable continuations when receiving the initial part of a melody, or software that can guess likely or reasonable completions of initial parts of a sentence or a text.

Some Related Publications

Buonamente, M., Dindo, H., Chella, A. and Johnsson, M. (2018). Simulating Music with Associative Self-Organizing Maps. Journal of Biologically Inspired Cognitive Architectures.

Buonamente, M., Dindo, H., Chella, A. and Johnsson, M. (2018). Simulating Music with Associative Self-Organizing Maps. To appear in the proceedings of BICA 2018, Prague, Czech Republic.

Johnsson, M. and Gil, D. (2011). Internal Simulation of Perceptions and Actions. In From Brains to Systems: Brain-Inspired Cognitive Systems 2010, 87-100, Springer, ISBN 978-1-4614-0163-6.

Johnsson, M., Balkenius, C. and Hesslow, G. (2011). Multimodal System Based on Self-Organizing Maps. In Madani, K., Correia, A.D., Rosa, R., and Filipe, J. (Eds.) Studies in Computational Intelligence, 343, 251-263, Springer, ISBN 978-3-642-20205-6.

Johnsson, M., Gil, D., Hesslow, G. and Balkenius C. (2011). Internal Simulation in a Bimodal System. In Kofod-Petersen, A., Heintz, F., and Langseth, H. (Eds.) Proceedings of the Scandinavian Conference on Artificial Intelligence (SCAI) 2011, Trondheim, Norway, IOS Press, 173-182. ISBN: 978-1-60750-753-6.

Johnsson, M., Gil, D., Balkenius, C. and Hesslow, G. (2010). Supervised Architectures for Internal Simulation of Perceptions and Actions. In Hernandez, C., Gomez, J., and Sanz, R. (Eds.) Proceedings of Brain Inspired Cognitive Systems (BICS) 2010, Madrid, Spain. ISBN 978-84-614-1870-1 / 978-84-614-1869-5.

Johnsson, M., Balkenius, C., and Hesslow, G. (2009). Neural Network Architecture for Crossmodal Activation and Perceptual Sequences. Papers from the AAAI Fall Symposium (Biologically Inspired Cognitive Architectures) 2009, Arlington, Virginia, USA. 85-86.