Although the Matsuo-Iwasawa Laboratory (hereinafter Matsuo Lab) is well known for promoting research on artificial intelligence (AI), many people may not know that it also focuses on robotics research.
We interviewed Mr. Tatsuya Matsushima, who has been active in Matsuo Lab’s robotics research since he was a master’s student, and Mr. Yusuke Iwasawa, a Matsuo Lab lecturer and adviser for the robotics team (the titles are at the time of interview. Mr. Iwasawa is now associate professor since January 2024)
This article will introduce their purpose, history, overall picture of activities, and future prospects of the Matsuo Lab’s robotics research.
Promoting robot learning using deep learning at Matsuo Lab, where the goal is to “realize intelligence.”
ー Why does Matsuo Lab promote robotics research?
Matsushima:
The Matsuo Lab promotes robotics research because we believe that we can approach the goal of the Matsuo Lab, “unravel the mysteries of intelligence” by using actual machines that interact with the real world.
Through implementing robots as embodied systems and data acquisition of their operation, we aim to create “intelligent” behaviors that are multi-purpose and adaptable. This corresponds to the research on world models that Matsuo Lab has been focusing on in recent years, using real-world data.
ー What exactly is the robotics research that you are conducting?
松嶋：
Matsushima：
The robotics research team (TRAIL) promotes the following three activities: research activities, lectures, and robotics club management.
Our research activities focus on robot learning, which aims to learn flexible environmental understanding and control of robots from data using machine learning techniques. Until now, robots have been utilized in situations such as factories, where they are precisely designed and perform a predetermined task with high speed and precision in an environment where their movements are constrained. On the other hand, at Matsuo Lab, we are developing technologies to improve generalization performance for various situations and adaptability to unknown environments by using advanced AI technologies for various tasks in more challenging and diverse environments closer to humans, such as home and daily environments.
For example, a robot that repeats the same action of “tightening a nut” is commonly used in factories, but a cleaning robot used at home will not work in the same way. This is because even if the robot perfectly understands the structure of house A and learns to behave appropriately in that environment, the structure of house B will be different, so the robot’s programming and prior knowledge are not applicable. This unique aspect of the home and retail environments requires a behavior appropriate to each environment. To overcome these problems, we focus on “generalization” and “adaptation”.
Therefore, Matsuo Lab conducts research on reinforcement learning and imitation learning algorithms, which aim to acquire better robot behavior by learning from the data of the robot’s previous movements. We are also studying data-driven robotics systems that combine deep learning techniques to obtain high generalizability and adaptability to various objects, layouts, and tasks that may be encountered in the home. We are also researching the construction of these data-driven robot systems. In addition, I have been working on simulation techniques (transfer from simulator to real world and simulator learning using real-world data) to efficiently collect data for learning, using data obtained from actual robots and simulators that simulate them.
In addition to the above research activities, I have also given lectures and managed a robotics club to increase the number of colleagues interested in combining robotics and artificial intelligence and promoting research.
ー The robotic team at Matsuo Lab conducts a wide range of systematic activities but how did the research begin?
Iwasawa:
Although our activities using robots began about four years ago, Dr. Matsuo has been repeating for some time that “robotics research is important.
From the aspect of intelligence research, studying robots (or interacting with the environment) is essential, as the word “physicality” is often used. Also, from the aspect of applications, it is vital to cross Japan’s strength in hardware creation, including robots, with the ever-evolving AI technology. In the digital world, research can proceed as long as data can be obtained, and since it proceeds according to English standards, it is difficult for non-English-speaking Japan to compete on a global scale. On the other hand, with its technological prowess, Japan would have an easier time gaining an advantage in the physical world, since it involves hardware.
However, the hardware needs to be more refined and easier to operate. Most machine learning researchers cannot operate a robot. Of course, you can operate if you do things right, but the unique rules create high barriers for new entry. It would be a good opportunity for us to promote robotics research in such a situation.
In this vein, we initially started robotics research on a small scale with a few researchers. Then, around 2018, we started robotics research in earnest when Matsushima, who strongly desired to conduct R&D using actual machines, joined us.
Matsushima:
Around 2019, we rented a dedicated floor for robot experiments in a building near the Akamon Gate and began full-scale research using actual machines. Initially, it was just a plain room with only one robot and a desk. Four years later, multiple robots and equipment have been installed, the number of people involved has increased significantly, and the facilities have been enhanced. （As of February 2023, we have three Toyota HSR mobile manipulators, six robot arms (two Manufactory xArm 7, two Ufactory lite 6, one Sawyer, and one other), two Unitree Go1Edu quadruped robots, seven Roomba robots, and many other actual machines that we can operate freely.
▼Before
Robot room with no environment when we first moved in.
▼After
We have about 30 members involved in robotics research and conduct research and study sessions day and night in an environment where they can freely operate a large number of actual machines.
Starting research from scratch and spending days going through trial-and-error to realize the need to grasp the intuition of robotics research.
Matsushima:
It sounds like smooth sailing when summarized in this way, but looking back on my actions at the time, I have many things to reflect on. It was good that I started my research on robot learning with great vigor, but every paper on robotics-related projects I did in my first year failed to make it to international conferences. Around the first half of 2019, I realized that I needed to understand the intuition and difficulties of robotics.
After those experiences, I arrived at the point where we needed to think about how to do it. I remember that we made various plans, such as increasing the number of staff and developing through discussions. At that time, we started to think that we should participate in robot contests, build an actual system once, and increase the number of people who participate in robot contests because it is very difficult to participate.
I was completely struggling my way through the process with no knowledge of robotics R&D “etiquette,” so I still have bitter memories of a professor specializing in robotics, who is now a close friend of mine, pointing out that “Matsuo Lab should learn how to write entry sheets from other teams” in a very rudimentary way (laugh). I still have a bitter memory of this (laughs).
Iwasawa:
Compared to other research, it’s a lot more challenging to develop. Only a few people can do both machine learning and deep learning. During that period, I realized that it would be quite difficult to conduct normal research on robots without a well-developed research system.
Matsushima:
From there, we started earnestly preparing for the WRS (robot competition).
Leading lectures and launching of robotics club with the aim to discover and nurture human resources.
ー What else did you do to establish a research system?
Matsushima：
Since we needed a large number of people to participate in the competition, we also focused on finding and training human resources to strengthen the robotics team structure.
One of the first initiatives was to plan and conduct lectures on areas related to robot learning.
Since there were many cases in which people interested in reinforcement learning were also interested in robotics, we first held a lecture on reinforcement learning to broaden the scope of our research by communicating the latest research findings to a wide audience.
For example, as lectures, we have held a total of six intensive seminars dealing with advanced topics on deep reinforcement learning once or twice a year since 2020, as well as lectures on reinforcement learning and physical simulation and graphics in the “World Models and Intelligence” course. In addition, we offer a project exercise-type lecture called “Artificial Intelligence Application Project” so that undergraduate and graduate students who are not assigned to a laboratory can work on a project using an actual robot, obtaining hands-on experience.
Another initiative is the “AI and Robotics Circle TRAIL,” which we have been running since 2022.
This is a robotics club that Komaba students can join. We intend to increase the number of people who can get involved when their motivation is at its highest, in their first and second years of undergraduate school.
Only some people were interested in robots when they first entered the University of Tokyo. There are robotics clubs, but there is no place where people who want to learn machine learning can quickly get involved. So, if there were a place where Komaba students could casually use robots, it would attract more people. This was the seed of the idea for TRAIL. If I were a first-year undergraduate student, I would have jumped at the idea and always wanted to create such a place.
With this background, we have created an environment where students can get involved in robotics research from the undergraduate level. TRAIL, an AI and robotics club, provides an environment where students can gain experience in developing robot systems to students and aims to participate and win prizes in the RoboCup world championship. In addition, TRAIL is developing “Introduction to Robot Systems” (a.k.a. RoombaHack) as open-source educational material, aiming to create a world where anyone can easily gain experience developing AI using real robots.
Right now, we have about 15 first-year undergraduate students participating in the program who voluntarily work tirelessly with the robot until late every day, and we are overwhelmed by their enthusiasm.
From the Web to DeepLearning and Robotics Research.
In the future, the connection with the “world model,” which is basic research, will be in full swing.
ー After all these years, where do you see the robotics team in 2023?
Iwasawa:
I feel that we have finally completed the most essential part of the groundwork for advancing our research. I think that the groundwork has been laid to achieve great progress in the future.
Matsushima:
Looking back, all we have done so far is the foundation work (laughs).
Iwasawa:
There may still be holes in the ground, though (laughs).
ー Now that the most important foundation work is finished, what are some of the major changes?
Iwasawa:
That would be the connection to the “world model” being researched in the Matsuo Laboratory. Recently, we have also been working on creating robot demos that apply the results and findings from our basic research to robots. In Matsuo Lab, both people are interested in basic technologies, such as algorithms for making smart robots, and people who want to make robots do helpful tasks. I think it is rare to find a case like Matsuo Lab where both can work together. When a person researching a technology makes a specific contribution, the robotics team can evaluate its real-world application. By connecting and creating such an environment, we are finally realizing a system that returns the accuracy and evaluation of the actual application of the robot to the research side.
Matsushima:
Another important point is that when we set a goal of creating a world in which autonomously operating robots are integrated into daily life and, in the process, elucidating the mechanism of “intelligence” itself, we now have a better understanding of the difficulties involved in actually doing it.
In many cases, it is only after applying machine learning research to the actual robotics field that we realize the problem set up in the first place is different from the reality of the situation. This challenge can only be faced by a laboratory that conducts both robotics and machine learning research, and it is a great challenge.
Iwasawa:
Since research cannot be conducted unless the robot is in operation, the increase in the number of actual robots has made it possible to see whether the robot is moving properly. In the field of research, there is competition over small differences in result numbers, but in robots, you can see whether the robot is actually moving, so it is genuinely fun research.
Matsushima:
Robotics research is an area where we put all our efforts into making the robot move, so it is easy to understand that we can evaluate whether or not the robot is moving properly.
ー From your point of view, what kind of laboratory do you think Matsuo Lab is?
Iwasawa:
Suppose you have a goal you want to achieve. In that case, Matsuo Lab is an organization that moves forward without falling into the stereotype of so-called “university laboratories” when it comes to achieving it.
Matsushima:
I completely agree. Whenever I say, “I want to do something like this,” or “I think this is important,” the culture encourages me to “give it a try,” and if I have a problem, I can easily ask for advice. That’s why I could express my desire to do it and move forward with robotics research, even though I was not working on robots in earnest before I joined the company.
However, I am one of many doing this kind of thing. When advancing my robotics research, I was very much referring to the Matsuo Lab’s shift from web research to deep learning research. I am using that method as a reference and working on a robot version.
Also, the Matsuo Lab has a strong culture of actively involving the people around us, so we were able to start our robotics research, which requires scale, on a certain scale from the beginning. However, we are now accumulating internal know-how, partly because we have staff who understand the hardware and partly because we were able to take over the use of the actual robot equipment smoothly. I feel that internal know-how is accumulating and the achievements based on such a culture of involving others have encouraged newcomers to the laboratory to engage in robotics research smoothly.
Our goal is the coexistence of humans and robots.
We want to increase the points of contact between robotics research and society without confining ourselves to the laboratory.
ー Can I ask you about your future prospects in robotics research?
Matsushima：
I would like to consider how the technology will be connected to services when it is introduced to the world and the mechanism itself. I don’t think there are currently any situations where robots are operating freely in our daily lives. Still, I believe the technological elements are in place to make that possible to some extent.
It would be great if we could do something like robot development that is not confined to the laboratory.
In this regard, I would like to create opportunities to interact with various people in society regarding robot technology, not only through basic research on robots and artificial intelligence but also through participation in competitions such as robot contests and public demonstrations, and feed this back into our research activities.
Last December, we held an advent calendar open to outside participants. Our robot research group, TRAIL will participate in RoboCup Japan and RoboCup World Championship, mainly consisting of undergraduate students. We will also demonstrate our robot at the May Festival at the Hongo Campus of the University of Tokyo for the general public, so please come and visit us.
(updated on April, 2024)
TRAIL’s tidy-up robot system won first place at the RoboCup23 (Japan) and third place in the RobotCup23 (world competition).
ー Thank you very much. Lastly, please tell us about the kind of person you would like to do research with.
Matsushima：
I would like the candidate to be willing to use technology to solve the problems ahead. I want to promote research and development with someone who is quick to move that way because I need to get my hands on it, not just think about it.
When I talk with Mr. Iwasawa, I often refer to him as a “hacker-like” person. Many people in Matsuo Lab try to “hack” by grasping the structure of issues and getting their hands on them, and I think a likely-minded candidate will enjoy the lab environment. If you are interested, please visit the lab.
twitter: https://twitter.com/trail_ut
site: https://trail.t.u-tokyo.ac.jp/ja/
（Profile）
Yusuke Iwasawa / Lecturer at Matsuo Laboratory of the University of Tokyo (as of the time of the interview, he is now an associate professor)
Biography
- March, 2014: Graduated from Graduate School of Science and Engineering, Sophia University with a major in informatics
- March, 2017: Graduated from Graduate School of Engineering, The University of Tokyo
- April, 2017 ~ June, 2018: 年4月〜2018年6月 Specially appointed researcher Fellow, Graduate School of Engineering, University of Tokyo
- July, 2018 ~ November, 2020: Specially Appointed Assistant Professor
- December, 2020 ~ March, 2022: Specially Appointed Lecturer
- April, 2022 ~ December, 2023: Lecturer (as of the time of interview)
- January, 2024 ~ until now: Associated Professor (updated in May 2024)
Area of specialty
- Transfer learning, deep representation learning, and deep learning applications
Awards
- Best Student Paper Award at Web Intelligence 2019，
- National Conference on Artificial Intelligence Excellence Award, National Conference on Artificial Intelligence Student Incentive Award
- MIRU Excellence Award, etc.
Other activities
- Lecturer of “Fundamental of Deep Learning”, “AI Management Endowed Chair”, “World Models and Intelligence”, etc.
- Supervised translation (translation coordination) and shared translation of “Deep Learning”
Tashiya Matsushima/ Doctoral Student, Matsuo Laboratory, Department of Technology Management Strategy
Biography
- May 2018: Graduated from Intelligent Social Systems Course, Department of Systems Innovation, Faculty of Engineering, The University of Tokyo
- Mar 2020: Completed Master’s program in the Department of Management and Strategy of Technology, Graduate School of Engineering, The University of Tokyo (Awarded the Dean’s Prize)
- April 2020~: Doctoral Student, Department of Technology Management and Strategy, Graduate School of Engineering, The University of Tokyo
- Student Editorial Board Member of the Japanese Society for Artificial Intelligence since 2018
Area of Specialization
- Modeling of the dynamics of the environment using deep genera
