Paper 1
■書誌情報
Xiaojie Gu, Sherry T. Tong, Aosong Feng, Sophia Simeng Han, Jinghui Lu, Yingjian Chen, Yusuke Iwasawa, Yutaka Matsuo, Chanjun Park, Rex Ying, Irene Li: Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models, Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), Findings, October 2026
■概要
Large language models (LLMs) are often evaluated based solely on final-answer accuracy, which can obscure failures occurring at intermediate reasoning steps. To address this limitation, we introduce Omanic, an open-domain 4-hop question answering benchmark designed for fine-grained, step-wise evaluation of multi-hop reasoning. Omanic consists of 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed, human-annotated evaluation examples (OmanicBench). Each evaluation question is decomposed into single-hop sub-questions, intermediate answers, and structured reasoning graph topologies, enabling detailed diagnosis of where reasoning breaks down. Experiments with both proprietary and open-source LLMs reveal several characteristic failure patterns, including a later-hop bottleneck, a factual knowledge floor, and error propagation along reasoning chains. Furthermore, fine-tuning on OmanicSynth transfers effectively to six reasoning and mathematics benchmarks, achieving an average improvement of 7.41 points and demonstrating its utility as supervision for improving general reasoning capabilities.
Paper 2
■書誌情報
Kohsei Matsutani, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo: Zipping the Thought: When and How Compressed Reasoning Data Works in LLM Post-Training, Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), Findings, October 2026
■概要
Large language models (LLMs) can now solve complex problems through long chain-of-thought (CoT) reasoning, but the trade-off between performance and token cost remains a central challenge. To address this issue, supervised fine-tuning (SFT) often uses compressed reasoning data, where CoT traces are shortened into compact forms. However, the effect of such compressed reasoning data on post-training remains poorly understood. In this paper, we propose a taxonomy of CoT consisting of Explicit CoT, which outputs all operations without aggregation, Composed CoT, which combines multiple operations into a single step, and Implicit CoT, which omits intermediate operations. We construct a synthetic compositional reasoning task that allows controlled variation of difficulty, compression granularity, and data size, and conducted a comprehensive set of experiments across different model families and sizes. Notably, we find that (i) coarser CoT requires more SFT data, (ii) compared with Explicit CoT, Composed CoT and Implicit CoT benefit more from data scaling, while Composed CoT benefits from data repetition and Implicit CoT tends to lead to memorization, (iii) unlike SFT, subsequent reinforcement learning (RL) with verifiable rewards (RLVR) decomposes compressed steps learned during SFT, and (iv) unidirectional CoT ordering shows stronger generalization on longer sequential tasks. Our findings provide implications for CoT design under data resource constraints and offer important insights into the mechanisms of SFT and RL in LLM post-training.
Paper 3
■書誌情報
Koshiro Aoki, Ryota Takatsuki, Gouki Minegishi, Yusuke Haruki, Daisuke Kawahara: In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?, Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), Findings, October 2026
■概要
Whether large language models (LLMs) can control their own internal representations matters for both machine metacognition and AI safety. A recent study applied neurofeedback to LLMs and claimed that they can control their internal representations. However, the reported control may rely on superficial mechanisms rather than genuine internal access because the control targets in that study are not privileged, meaning that a third party can infer them from the prompt. We redesign the neurofeedback paradigm for LLMs so that the control target satisfies the privileged access requirement, which is closer to neurofeedback experiments in human cognitive neuroscience. Under this stricter setting, the models do not demonstrate reliable control over privileged internal representations, suggesting that previously reported control cannot exclude the possibility that it relies on superficial mechanisms. Our results indicate that rigorous assessments of metacognition in LLMs require evaluation methods that demand privileged access.
Paper 4
■書誌情報
Shota Ekuni, Jie Zeng, Yohei Kobashi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo: Training Lightweight PII Text Detectors via Rubric-Guided Contrastive Data Synthesis, Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), Industry Track, October 2026
■概要
The widespread use of large language models (LLMs) has increased the risk of leaking personally identifiable information (PII), making PII detection increasingly important. PII detection often needs to process large volumes of text quickly and accurately, so training a lightweight classifier is a practical choice. However, training such a classifier requires a large amount of annotated text containing PII, while privacy-sensitive real text is difficult to collect, share, and redistribute. We therefore propose a method that induces rubrics (label-decision criteria) for separating Sensitive from Non-sensitive text from a small set of labeled seed examples, generates contrastive synthetic training data based on these rubrics, and trains a RoBERTa classifier. Experiments on the SPeDaC1 and SPY datasets show that our method outperforms simple synthesis baselines across all conditions of the two datasets (SPeDaC1 and SPY) and the two generator models (Qwen3.5-9B and Gemma-4-26B-A4B-NVFP4). By requiring only about ten labeled examples in the target domain, our method offers practical value in generating training data for a PII detection model suited to that domain.