GSoC 2015 BBOB

Performance evaluation metrics for evolutionary algorithms are crucial component for parameter tuning, testing new algorithm and problems, as well as characterizing and explaining the properties and behavior of algorithms by statistical runtime comparisons. One such performance evaluation testbed is the Real-Parameter Black-Box Optimization Benchmark (BBOB). The aim of this project is adding a new module to pagmo::util containing several performance evaluation methods from BBOB 2015 experimental setup (see [1] and [2]). The completed module will allow the users to run vast benchmarking campaigns, and analyze the performance of algorithms and initial parameters over a large sets of problems. Student is expected to get familiar with the performance evaluation methods of evolutionary algorithms (with the help of the mentor)

[1] Real-Parameter Black-Box Optimization Benchmarking: Experimental Setup. PDF with explanation and source code

[2] Nikolaus Hansen and Anne Auger. "Performance evaluation of an advanced local search evolutionary algorithm". In In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE Press, 2005 ONLINE

Implementation Details:

The student will extend the pagmo::util module with the pagmo::util::bbob module (as well as interface it to PyGMO's PyGMO.util.bbob). Where possible, the existing reference code should be reused and wrapped as C++ classes. Second part of the project will be dedicated to enriching PyGMO with results visualization (using matplotlib) and writing interface for a more extensive experimental setups (e.g. large benchmarking campaigns over many problems and algorithms).

An example for individual algorithm comparison:

from PyGMO import *
from PyGMO.util import bbob

ert = bbob.compute_ert(algorithm.jde(variant=2, gen=20), problem.rosenbrock(10), n_runs=1000)  # Compute the expected running time data
bbob.plot_cumulative_distribution(ert)  # Presents the results on a cumulative distribution plot

Required programming skills: Python , C++

Skills that help: Knowledge of statistical testing, matplotlib and data presentation

Mentors: Mentors#Krzysztof, Mentors#Dario