智源LIVE 第57期｜开放世界自主智能体的构建 Building Autonomous Agents in Open World

​

开发能够在复杂领域执行各种任务的智能体是实现具备通用能力的人工智能的关键步骤。JARVIS-1是一个基于视觉语言基础模型的智能体，能够从多模态用户和环境输入中生成适用于长期任务的计划，并将其转化为在《我的世界》中的鼠标和键盘控制。我们为JARVIS-1配备了一个多模态memory模块，使JARVIS-1可以利用预训练的知识和在实际游戏生存中收集的经验进行规划。在使用和人类一致的视觉观察空间和动作空间下，JARVIS-1能够从头完成开放世界游戏《我的世界》下的200多个任务，包括短期的“砍树”到长期的“合成钻石镐”等。随着JARVIS-1与环境交互时间的增长，我们还观察到了Agent持续的改进，尤其是在完成更复杂的任务方面。

在这次演讲中，我们首先会介绍如何在像Minecraft这样的开放世界中构建指令跟随代理。然后，我们将确定当扩展到长期任务时，开放世界代理面临的挑战以及如何利用预训练基础模型来解决这些问题。最后，我们将展示在Minecraft上进行的大量实验来验证代理的能力。

Developing advanced agents that can perform a wide range of tasks in complex domains is a crucial step towards achieving artificial intelligence with general capabilities. We introduce JARVIS-1, a multimodal language model based agent that can robustly produce plans for long-horizon tasks from multimodal user and environment inputs, and translate them into motor control in Minecraft, a popular yet challenging open-world testbed for generalist agents. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. JARVIS-1 is capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., “chopping trees” to long-horizon tasks, e.g., “obtaining a diamond pickaxe”. 

In this talk, we will first introduce how to build instruction-following agents in open world like Minecraft. Then we will identify challenges for open-world agents when scaling up to long-horizon tasks and how to leverage pre-trained foundation models to address it. Finally, we will present extensive experiments conducted on Minecraft to validate the agent’s abilities.

王子豪，北京大学智能学院博士生，导师为梁一韬教授。曾获国家奖学金、北京市优秀毕业生等荣誉。主要研究方向为开放世界下多任务智能体的构建，尤其关心基于基础模型的智能体的泛化能力。近年来在CVPR、NeurIPS等人工智能顶会上发表多篇论文，曾获ICML研讨会最佳论文奖。担任ICML、NeurIPS、ICLR等多个国际机器学习会议审稿人。

Zihao Wang is a Ph.D. student at Peking University’s School of Intelligence Science and Technology, under the guidance of Prof. Yitao Liang. His research focuses on developing open-world embodied agents with multi-task abilities, including visual localization, task planning, and decision making. He is particularly interested in utilizing large pre-trained Foundation Models to enhance the agent’s generalization capabilities. In recent years, he has published multiple papers at top artificial intelligence conferences such as CVPR and NeurIPS. He has also won the Best Paper Award at the ICML workshop. He has served as a reviewer for various international machine learning conferences including ICML, NeurIPS, and ICLR for a long time.

主页地址：https://zhwang4ai.github.io

蔡少斐目前是北京大学人工智能研究院二年级博士生，导师是梁一韬教授。在此之前，他分别于西安交通大学和中国科学院计算技术研究所获得学士、硕士学位。他的研究兴趣主要包括决策大模型、语言大模型以及游戏智能。他已在 CVPR 、NeurIPS 等人工智能顶会上发表过多篇论文，并专注于开放世界下智能体决策控制研究。担任 NeurIPS 、 ICLR 等国际学术会议审稿人。

Cai Shaofei is currently a second-year doctoral student at the Institute of Artificial Intelligence, Peking University, under the supervision of Professor Liang Yitao. Prior to this, he obtained his bachelor’s and master’s degrees from Xi’an Jiaotong University and the Institute of Computing Technology, Chinese Academy of Sciences respectively. His research interests mainly include decision-making models, language models, and game intelligence. He has published multiple papers at top artificial intelligence conferences such as CVPR and NeurIPS, focusing on research in intelligent agent decision control in open worlds. He serves as a reviewer for international academic conferences such as NeurIPS and ICLR.

主页：https://phython96.github.io/

​Read More