Current browse context:
cs.AI
Change to browse by:
References & Citations
Computer Science > Artificial Intelligence
Title: Abrupt and spontaneous strategy switches emerge in simple regularised neural networks
(Submitted on 22 Feb 2023 (v1), last revised 1 Mar 2024 (this version, v4))
Abstract: Humans sometimes have an insight that leads to a sudden and drastic performance improvement on the task they are working on. Sudden strategy adaptations are often linked to insights, considered to be a unique aspect of human cognition tied to complex processes such as creativity or meta-cognitive reasoning. Here, we take a learning perspective and ask whether insight-like behaviour can occur in simple artificial neural networks, even when the models only learn to form input-output associations through gradual gradient descent. We compared learning dynamics in humans and regularised neural networks in a perceptual decision task that included a hidden regularity to solve the task more efficiently. Our results show that only some humans discover this regularity, whose behaviour was marked by a sudden and abrupt strategy switch that reflects an aha-moment. Notably, we find that simple neural networks with a gradual learning rule and a constant learning rate closely mimicked behavioural characteristics of human insight-like switches, exhibiting delay of insight, suddenness and selective occurrence in only some networks. Analyses of network architectures and learning dynamics revealed that insight-like behaviour crucially depended on a regularised gating mechanism and noise added to gradient updates, which allowed the networks to accumulate "silent knowledge" that is initially suppressed by regularised (attentional) gating. This suggests that insight-like behaviour can arise naturally from gradual learning in simple neural networks, where it reflects the combined influences of noise, gating and regularisation.
Submission history
From: Anika Theresa Löwe [view email][v1] Wed, 22 Feb 2023 12:48:45 GMT (2219kb,D)
[v2] Sat, 15 Jul 2023 06:24:33 GMT (1279kb,D)
[v3] Wed, 4 Oct 2023 13:30:40 GMT (1279kb,D)
[v4] Fri, 1 Mar 2024 16:54:07 GMT (1316kb,D)
Link back to: arXiv, form interface, contact.