Bibliographic Information
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, Yutaka Matsuo. “On the Multilingual Ability of Decoder-based Pre-trained Language On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons”. 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2024)
Abstract
Current decoder-based pre-trained language models (PLMs) successfully demonstrate multilingual capabilities, but it is unclear how We analyze neuron-level internal behavior of multilingual decoder-based PLMs: The existence of neurons that fire “uniquely for We analyze neuron-level internal behavior of multilingual decoder-based PLMs: The existence of neurons that fire “uniquely for each language” within decoder-only multilingual PLMs. and Japanese, and show that language-specific neurons are unique with a slight overlap (< 5%) between languages and are mainly distributed in the This trend is consistent across various languages and models. We also tamper with less than 1% of the total neurons in each model during inference and show that tampering with few language-specific neurons drastically changes the probability of target language We also tamper with less than 1% of the total neurons in each model during inference and show that tampering with few language-specific neurons drastically changes the probability of target language occurrence in text generation during inference.
