Short-Term Scientific Mission

Hardware-Independent Benchmarking

Main theme: Benchmarking
Grantee: Marie Anastacio, RWTH Aachen University, Aachen, Germany
Host: Théo Matricon, INRIA Rennes, Rennes, France

Start date: 2025-09-01
End date: 2025-09-12
Awarded: 2025-06-30
Report approved: 2025-10-31

Description

This work focuses on developing a hardware-independent benchmarking tool for optimisation algorithms, enabling researchers to efficiently assess their methods without relying on identical execution environments. Building on prior work in statistical performance comparison, the project aims to integrate techniques that account for variability in runtime environments. During the visit to INRIA Rennes, the applicant will collaborate with the DiverSE team to provide developers with rapid, statistically grounded feedback on algorithm performance, contributing to best practices in the optimisation community.

Achievements

This STSM successfully developed a framework for a hardware-independent benchmarking tool. During the visit, a database was established, capturing key variability features such as hardware specifications, software configurations, and operating environments, using data from recent SAT competitions (2022–2024). A flexible architecture was designed to support performance prediction models, allowing integration of diverse approaches from both automated algorithm configuration and software engineering communities. The tool’s design facilitates rapid, statistically robust feedback for developers, with plans to present it to the community for further refinement. Collaborative discussions also paved the way for future research at the intersection between software engineering and automated algorithm design.