松尾・岩澤研究室では，「知能を創る」というミッションのもと、世界モデルをはじめとした深層学習やそれを超える基礎技術の開発、ロボティクスや大規模言語モデル、アルゴリズムの社会実証といった幅広い研究領域で活動しています。
こうした活動を更に拡大するため、リサーチインターンシップを開催し、15名の方にご参加いただきました。
▼リサーチインターンシップ概要
https://weblab.t.u-tokyo.ac.jp/news/20240417/
▼インターンテーマ／メンターの紹介記事
https://weblab.t.u-tokyo.ac.jp/news/20240426/
本記事では、リサーチインターンに参加いただいたメンバーの体験記をご紹介します。
About Me
Hi, I’m Meiyun Wang, a third-year Ph.D. student in the Faculty of Engineering at the University of Tokyo. My research focuses on NLP & LLM applications. While I spend a lot of time working independently, I’m always excited to collaborate and exchange ideas with other researchers in the field. This passion led me to join the internship program at Matsuo Lab, where I had the chance to dive into some fascinating work.
About the Research
During my internship, I worked on a project titled “Lost in the Distance: Large Language Models Struggle to Capture Long-Distance Relational Knowledge,” under the guidance of Professor Kojima. Before settling on this topic, we spent a lot of time exploring the limitations of large language models.
For example, one well-known issue is that LLMs often fail to generalize. If a model is trained on a sentence like “A is B,” it usually can’t infer the reverse—“B is A.” This is called “The Reversal Curse.” Another limitation is that LLMs tend to perform best when important information is at the beginning or end of the input context. But if the key details are buried in the middle of a long text, even models designed for long-context tasks struggle. This is known as the “Lost in the Middle” phenomenon.
Building on these findings, we decided to dig deeper into relational extraction tasks, like causal reasoning, to see if LLMs show similar patterns. After running a bunch of experiments, we discovered something new: the “Lost in the Distance” phenomenon. This happens when LLMs have trouble capturing relationships between pieces of information that are separated by noise.
To test this, we designed an experiment where we added artificial noise between two related elements and measured how well the models performed as the distance between them grew. The results were pretty clear: while LLMs can handle a little noise without too much trouble, their ability to reason about distant relationships drops sharply as the noise increases. This trend held true for both forward and backward prediction tasks.
We tested this across several models (GPT-4, Gemini-1.5-Pro, GPT-4o-Mini, Gemini-1.5-Flash, and Claude-3.5-Sonnet) and tasks (causal reasoning and knowledge extraction). The results highlight a big limitation in how LLMs process relational knowledge over long contexts. I’m thrilled to share that our research was accepted at NAACL 2025!
About the Experience
This internship was a bit of a whirlwind—I was juggling other research projects at the same time, so time was tight. From finalizing the topic to submitting the paper, the whole process took about two weeks. It was intense, but we managed to run a ton of experiments and tweak our setups to make the findings as solid as possible.
I’m incredibly grateful to Professor Kojima for his guidance. He was amazing at spotting the little details and helping us refine our experiments, even with such a tight deadline. Thanks to him, we were able to pull everything together and get the paper submitted on time.
This internship was such a rewarding experience for me. Not only did I improve my research skills, but I also got to see how leading researchers think and approach problems. Before this, my research was more about experimenting with methods to improve model performance. But this experience taught me to think more deeply about the “why” behind the phenomena we observe.
I’m really thankful for this opportunity and for Professor Kojima’s support and encouragement. I also want to give a shoutout to the other professors at Matsuo Lab for their insightful feedback and advice throughout the internship. It was such a great learning environment, and I’m excited to take these lessons forward in my research journey!
いかがでしたでしょうか？
松尾研では研究員を積極的に募集しております。気になる方は下記をご覧ください！
https://weblab.t.u-tokyo.ac.jp/joinus/career/
