Towards Scientific Benchmarks: On Increasing the Credibility of Benchmarks

Gundersen, Odd ErikI. Pratikakis and M. Spagnuolo and T. Theoharis and L. Van Gool and R. Veltkamp2015-04-272015-04-272015https://doi.org/10.2312/3dor.20151059Problem: Increasing the credibility of results from scientific benchmarks. Goal: Specify what exactly is required in order for a benchmark to be scientific. Contribution: (i) Specification of what it entails for a benchmark to be scientific, (ii) a metric for measuring the replicability of an experiment, (iii) a metric for measuring the replicability of a set of experiments, and (iv) Analysis of the replicability of SHREC 2015. Result: Replicability of SHREC 2015 can be increased by open sourcing the methods compared and improving documentation.I.5.2 [Pattern Recognition]Design MethodologyTowards Scientific Benchmarks: On Increasing the Credibility of Benchmarks10.2312/3dor.2015105983-86