Evaluation

The two main goals of the TCHES 2021 artifact review process were to improve functionality and reusability of artifacts to enable reproducibility and extension by the scientific community.

Reproducibility, in the context of computational experiments, means that the scientific results claimed can be obtained by a different team using the original authors’ artifacts. The artifact review process for TCHES 2021 did not include attempting to reproduce the experiment and verify the scientific claims in the accepted paper. Rather, the artifact review process for TCHES 2021 aimed at ensuring sufficient functionality of the artifact to enable a research team to attempt to reproduce the results. The artifact review process for TCHES 2021 also aimed to improve reusability, meaning that the artifacts are not just functional, but of sufficient quality that they could be extended and reused by others.

For more information, please see the TCHES 2021 Call for Artifacts and the report of the artifact review chair.