Hybrid Multimodal Generative Artificial Intelligence Approach for Mathematical Problem-Solving with Figures

초록

Mathematical problem-solving with visual components remains a significant challenge for artificial intelligence (AI) systems. This is because conventional OCR pipelines often fail to extract critical structural cues such as axes, tick marks, and spatial relationships in diagrams or graphs. Despite recent advances in generative models, a cohesive framework that integrates visual perception with symbolic inference is still lacking. Therefore, this study proposes a lightweight hybrid pipeline that combines ColPali for vision-language understanding with open-source large language models (LLMs) such as LLaMA for symbolic reasoning. We evaluated our system on the MathVision dataset, leveraging its category-level statistics to measure the performance across diverse visual math tasks. Our method improves accuracy by up to 29.3% compared to standalone models, demonstrating the effectiveness of integrating visual structures into mathematical reasoning. © The Korean Institute of Intelligent Systems. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

키워드