Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
复制标题
DOI：
10.48550/arxiv.2303.04671
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Chenfei Wu;Sheng-Kai Yin;Weizhen Qi;Xiaodong Wang;Zecheng Tang;Nan Duan
中科院分区：
文献类型：
--
作者：
Chenfei Wu;Sheng-Kai Yin;Weizhen Qi;Xiaodong Wang;Zecheng Tang;Nan Duan
Chatgpt吸引了跨场兴趣，因为它提供了一个具有出色的对话能力和许多领域的推理能力的语言界面。但是，由于Chatgpt接受了语言培训，因此目前无法处理或生成视觉世界中的图像。同时，视觉基础模型（例如视觉变压器或稳定的扩散），尽管显示出良好的视觉理解和发电能力，但它们仅是具有一轮固定输入和输出的特定任务的专家。为此，我们构建了一个称为\ textbf {Visual Chatgpt}的系统，结合了不同的视觉基础模型，以使用户能够通过1）发送和接收语言，而是图像2）提供复杂的视觉问题或视觉问题编辑说明需要与多个AI模型合作的多个步骤。 3）提供反馈并要求校正结果。我们设计了一系列提示，以考虑需要视觉反馈的多个输入/输出和模型的模型，将视觉模型信息注入chatgpt。实验表明，Visual Chatgpt在视觉基础模型的帮助下为研究Chatgpt的视觉角色打开了大门。我们的系统可在\ url {https://github.com/microsoft/visual-chatgpt}上公开获得。
ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called \textbf{Visual ChatGPT}, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models. Our system is publicly available at \url{https://github.com/microsoft/visual-chatgpt}.