Short-Term Scientific Mission

A Generic Tool for Sustainable Benchmarking

Main theme: Benchmarking
Grantee: Markus Iser, Karlsruhe Institute of Technology, Karlsruhe, Germany
Host: Théo Matricon, INRIA Rennes, Rennes, France

Start date: 2025-09-01
End date: 2025-09-12
Awarded: 2025-06-30
Report approved: 2025-10-31

Description

The aim of this STSM is to develop a generic, extensible and data-driven benchmarking tool for evaluating optimisation algorithms, focusing on sustainability and large-scale experimentation. The plan is to integrate the host’s and applicant’s existing methodologies into a modular architecture that supports easy adaptation to diverse problem domains and facilitates the instantiation of core components with new methods. Planned activities include designing key components and evaluating the integration of expert-defined problem categories. The resulting open-source tool will improve access to advanced benchmarking techniques and facilitate future advancements in solver evaluation and algorithmic competitions.

Achievements

A modular benchmarking framework was designed and prototyped to unify and extend existing tools for evaluating solvers. The architecture, data model, and adapters were developed to ensure extensibility, reusability, and integration with diverse infrastructures. The outcomes contributed to COST Action WG6 objectives by advancing modular, data-driven evaluation methods and fostering knowledge exchange. The framework was shown to improve accessibility, reproducibility, and collaboration in benchmarking research. Plans were made for further validation in competitions, joint student supervision, and publication of the tool and its empirical results.