MMPB: It's Time for Multi-Modal Personalization

Visual personalization is essential for user-facing AI systems, from smart homes to healthcare, where models must adapt to human-centric concepts and individual preferences. Yet, despite their broad capabilities, recent Vision–Language Models (VLMs) remain underexplored in this regard. We present MMPB, the first large-scale benchmark for VLM personalization, comprising 10k image–query pairs and 111 concepts across four categories (humans, animals, objects, characters). The human category further includes preference-grounded queries to test alignment with user-specific needs. MMPB evaluates personalization across three key task types—Awareness, Appropriateness, and Coherency—under a three-stage protocol: concept injection, multi-turn dialogue, and personalized querying. Using 23 widely adopted VLMs (both open- and closed-source), we show that most models struggle: they under-personalize, fail to maintain consistency across dialogue, and rarely exploit visual cues effectively. By surfacing these systematic limitations—from refusal behaviors to long-context forgetting—MMPB provides a scalable and rigorous foundation for advancing truly personalized multi-modal AI.