• Home
  • ニュース
  • EACL 2027 に当研究室の論文が採録
  • —

    EACL 2027 に当研究室の論文が採録

    ■書誌情報
    Yuya Imai, Kotaro Sakamoto, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo: The Probe Is Not the Handle: Decodable Targets and Causal Subspaces in VLM GUI Grounding, Proceedings of the 20th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2027, Main), March 2027

    ■概要
    GUI-grounding vision–language models (VLMs) map a screenshot and an instruction to click coordinates, and often miss. Yet the target is often readable inside the model: across four VLMs (three Qwen-lineage, one OLMo-lineage; 3–7B), a linear probe localizes it on 65–73% of failed clicks, well above a constant-location prior. But reading is not control: steering along or erasing the probe direction leaves the hit rate near baseline. A different 32-dimensional subspace found by Distributed Alignment Search—the handle, an effective but non-unique intervention channel—does control the click: interchange transfers a donor’s coordinate on all four, and erasure collapses the hit rate (partially on Molmo2). Writing the true coordinate into the handle repairs 44–74% of held-out failures on Qwen2.5-VL, versus under 10% for an equally large random-subspace edit; even a probe’s predicted coordinate, written the same way, repairs 37–59%. Routing with no test-time ground truth raises overall hit rate by 15–28 points on Qwen2.5-VL across three seeds, and by 8 and 20 points on Molmo2 and ScreenSpot-Pro, while lowering the hit rate on originally correct clicks by 3–43 points. For GUI agents, the results point to how representations are used, not only what they encode, as a lever for more reliable grounding.