From vehicle damage image recognition to medical record text semantic understanding, from satellite remote sensing data on farmland disasters to video analysis of building structures in property insurance, the data processing tasks facing the financial and insurance industries have long transcended mere text. These tasks span multiple modalities including images, text, and video, demanding exceptionally high capabilities in comprehensive information comprehension and reasoning. Multimodal artificial intelligence large language models offer a breakthrough solution to these challenges through their robust cross-modal comprehension and logical reasoning capabilities.

However, a critical question remains unresolved: How can we systematically evaluate the true capabilities of these models in insurance scenarios? To what extent can they actually address industry-specific multimodal tasks? 

For this purpose, Professor Xian XU, Fudan University and his team, in collaboration with scholars from University of Rochester, introduced The world's first multimodal model evaluation benchmark for the insurance industry, INS MMBench, marking a new breakthrough in the field of AI for Science (AI4S). 

The research team systematically organized multimodal tasks across the insurance value chain, establishing a hierarchical task framework and evaluation dataset. They conducted assessment tests on leading domestic and international multimodal large models such as GPT-4o, Qwen VL, and Gemini, delivering the industry's first quantifiable and reproducible evaluation framework for multimodal models. This framework clearly anchors the actual capabilities and application potential of multimodal AI within core insurance operations. This research has recently been accepted for presentation at ICCV 2025, a premier international conference in the fields of artificial intelligence and computer vision.

Translated by Ruihan CHEN

Full text in Chinese available at:

https://fddi.fudan.edu.cn/5b/43/c18985a744259/page.htm