Danushka先生が2/25（水）に松尾研を訪問されました。
松尾教授が所属していた石塚研究室にDanushka先生も所属されており、松尾教授とは古くから関わりがありました。そうしたご縁から、例年ご講演をいただいています。
当日は松尾研の研究員や配属学生以外にも、講義受講生などオンラインの参加者含め50名近くの方が参加し、講演いただきました。
Title: “The Map of Encoders — Navigating the Landscape of Sentence Encoders: A Quantum Relative Entropy Approach“
Abstract:
Representing the meaning of a sentence using a vector (embedding) is a fundamental task in numerous NLP applications.
A large number of sentence encoders have been developed following different training algorithms and fine-tuned on diverse tasks and datasets. For example, 17,515 sentence encoders are published in Hugging Face Hub (https://huggingface.co/models), and their global landscape remains unclear. This is a significant blocker when improving or selecting sentence encoders for NLP applications. Moreover, there is no universally agreed-upon metric for comparing sentence encoders. For example, models are often grouped by attributes such as their parameter size, developer, pre-trained data source, fine-tuned tasks, etc. in the Hugging Face Hub, while leader-boards such as MTEB (https://huggingface.co/spaces/mteb/leaderboard} group models by their downstream task performance. In this talk, I will describe our recent work on visualising sentence encoders at scale by creating a \textbf{map of encoders} where each sentence encoder is represented in relation to the other sentence encoders. Specifically, we first represent a sentence encoder using an embedding matrix of a sentence set, where each row corresponds to the embedding of a sentence. Next, we compute the Pairwise Inner-Product (PIP) matrix for a sentence encoder using its embedding matrix. Finally, we create a feature vector for each sentence encoder reflecting its Quantum Relative Entropy with respect to a unit base encoder. We construct a map of encoders covering 1101 publicly available sentence encoders, providing a new perspective of the landscape of the pre-trained sentence encoders. Our map accurately reflects various relationships between encoders, where encoders with similar attributes are proximally located on the map. Moreover, our encoder feature vectors can be used to accurately infer downstream task performance of the encoders, such as in retrieval and clustering tasks, demonstrating the faithfulness of our map.
Speaker Profile: ボレガラ・ダヌシカ氏は2009年に東京大学大学院情報理工学系研究科電子情報学専攻で博士課程（優秀学術研究成果に基づく短縮修了）を修了し，同研究科の助教，講師を得て，2018年から英国リバープール大学計算機学科教授として教育と研究活動を行っております．自然言語処理と機械学習を中心200本以上の論文としてその研究業績を発表しています．「深層学習」（近代科学社），「ウェブデータの機械学習」（機械学習プロフェッショナルシリース）など専門書を執筆しています．なお，株式会社アマゾン(Palo Alto, US)のAmazon Scholarとして深層学習の実世界への応用研究を指導しています．
Danushka先生、この度は松尾研に足をお運びいただきありがとうございました。
