CogAgent: A Visual Language Model for GUI Agents

Abstract

People are spending an enormous amount of time on digital devices through graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large language models (LLMs) such as ChatGPT can assist people in tasks like writing emails, but struggle to understand and interact with GUIs, thus limiting their potential to increase automation levels. In this paper, we introduce CogAgent, an 18-billion-parameter visual language model (VLM) specializing in GUI understanding and navigation. By utilizing both low-resolution and high-resolution image encoders, CogAgent supports input at a resolution of 1120*1120, enabling it to recognize tiny page elements and text. As a generalist visual language model, CogAgent achieves the state of the art on five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA, Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using only screenshots as input, outperforms LLM-based methods that consume extracted HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW, advancing the state of the art. The model and codes are available at https://github.com/THUDM/CogVLM.

사람들은 그래픽 사용자 인터페이스(GUI)를 통해 디지털 기기에서 엄청난 시간을 보내고 있습니다. 예를 들어, 컴퓨터나 스마트폰 화면 등이 있습니다. ChatGPT와 같은 대형 언어 모델(LLMs)은 이메일 작성과 같은 작업에서 사람들을 도울 수 있지만, GUI를 이해하고 상호 작용하는 데 어려움을 겪어 자동화 수준을 높이는 잠재력이 제한됩니다. 이 논문에서는 GUI 이해와 탐색을 전문으로 하는 180억 파라미터의 시각 언어 모델(VLM)인 CogAgent를 소개합니다. 저해상도 및 고해상도 이미지 인코더를 모두 활용하여 CogAgent는 1120*1120의 해상도에서 입력을 지원하며, 이를 통해 작은 페이지 요소와 텍스트를 인식할 수 있습니다. 일반적인 시각 언어 모델로서 CogAgent는 VQAv2, OK-VQA, Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, POPE를 포함한 다섯 가지 텍스트 중심 및 네 가지 일반 VQA 벤치마크에서 최고 수준의 성능을 달성합니다. 스크린샷만을 입력으로 사용하는 CogAgent는 추출된 HTML 텍스트를 사용하는 LLM 기반 방법보다 PC 및 Android GUI 탐색 작업에서 더 우수한 성능을 보여주며, Mind2Web 및 AITW에서 최신 기술을 더욱 발전시킵니다.

https://arxiv.org/abs/2312.08914

CogAgent: A Visual Language Model for GUI Agents

People are spending an enormous amount of time on digital devices through graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large language models (LLMs) such as ChatGPT can assist people in tasks like writing emails, but struggle to understand and interact with GUIs, thus limiting their potential to increase automation levels. In this paper, we introduce CogAgent, an 18-billion-parameter visual language model (VLM) specializing in GUI understanding and navigation. By utilizing both low-resolution and high-resolution image encoders, CogAgent supports input at a resolution of 1120*1120, enabling it to recognize tiny page elements and text. As a generalist visual language model, CogAgent achieves the state of the art on five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA, Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using only screenshots as input, outperforms LLM-based methods that consume extracted HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW, advancing the state of the art. The model and codes are available at \url{https://github.com/THUDM/CogVLM}.

arxiv.org

https://github.com/THUDM/CogVLM

GitHub - THUDM/CogVLM: a state-of-the-art-level open visual language model | 多模态预训练模型

a state-of-the-art-level open visual language model | 多模态预训练模型 - GitHub - THUDM/CogVLM: a state-of-the-art-level open visual language model | 多模态预训练模型

github.com