Recent advances have endowed Multimodal Large Language Models (MLLMs) with remarkable capabilities, enabling them to tackle complex challenges such as mathematical reasoning and multimodal understanding. Instead of concentrating solely on scaling model parameters during training, current research aims to strengthen inference-time reasoning. Techniques such as Self-Refinement, where models iteratively improve their output and engaging multiple models or agents in debate to achieve consensus have gained traction.

These methodologies heavily rely on the ability of MLLMs to evaluate and refine their responses. If such capability is not sufficiently secured, refinement might unintentionally impair performance, causing incorrect corrections and unnecessarily prolonged response times. Therefore, it is essential to investigate whether MLLMs can accurately identify and correct errors in their reasoning processes.

We introduce MMRefine, a MultiModal Refinement benchmark designed to evaluate the error refinement capabilities of Multimodal Large Language Models (MLLMs). With MMRefine, we can comprehensively analyze MLLMs' capability to detect and correct errors within a given initial solution across six distinct scenarios and six error types.