In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, consisting of both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

arxiv.org

Apple에서 MM1이라는 멀티모달 LLM에 대한 연구 결과를 공개하였습니다. (모델 코드나 가중치는 공개하지 않았고, 앞으로도 안 할 것 같습니다)Image Encoder와 VL-Connector, 그리고 데이터셋과 학습 방법 등에서 모델을 직접 학습하시거나 튜닝하시는 분들께서는 한 번쯤 살펴보셔도 좋을 것 같아 ChatGPT와 함께 정리한 내용을 공유합

news.hada.io