Knowledge Neurons

Knowledge neurons are individual neurons inside the feed-forward (MLP) layers of a pretrained transformer language model. A 2021 study proposed them as the storage locations for specific factual associations, such as the correct city for a country's capital. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei, working at Peking University and Microsoft Research, introduced the concept in "Knowledge Neurons in Pretrained Transformers" (arXiv:2104.08696, April 2021, later published at ACL 2022). Using BERT and a fill-in-the-blank cloze task, they proposed a knowledge attribution method that identifies the specific neurons whose activation correlates with a model expressing a given fact. They described the mechanism as resembling a key-value memory built into the MLP weights.

The paper's central practical claim was that, once a fact's knowledge neurons are located, the model's factual knowledge can be edited directly. A specific fact can be updated or erased by modifying the corresponding MLP weights, without any additional fine-tuning. This result helped motivate a wider line of research into model editing methods that locate and modify facts inside pretrained language models, treating factual recall as localized and directly manipulable rather than only changeable through retraining.

The knowledge neuron thesis has since been challenged. "What does the Knowledge Neuron Thesis Have to do with Knowledge?" by Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn (ICLR 2024, arXiv:2405.02421) found that the same neuron-editing methods used to locate and change factual associations can also edit unrelated syntactic patterns. Factual edits made this way often fail stricter consistency checks, such as producing the same answer under symmetric rephrasings or synonym substitutions. The authors argue that MLP weights more plausibly encode general, reusable patterns of token expression rather than discrete, addressable pieces of "knowledge." This makes successful editing an unreliable guide to where, or whether, knowledge, as opposed to a broader linguistic pattern, is actually stored.