Evaluation

Badges

New in 2024, authors could choose to have their artifacts evaluated by the CHES Artifact Evaluation Committee (AEC) against three badges: Artifacts Available, Artifacts Functional, and Artifacts Reproduced. Each evaluation is optional. This system broadly follows the conventions established in recent years in security research conferences such as USENIX Security and NDSS.

  1. ✅ IACR CHES Artifacts Available: To earn this badge, the AEC must judge that artifacts associated with the paper have been made available for retrieval. Other than making the artifacts available, this badge does not mandate any further requirements on functionality, correctness, or documentation. This is intended for authors who simply wish to make some supplementary material available that supports their paper. Examples include data sets, large appendices, and other documentation.
  2. 💡 IACR CHES Artifacts Functional: To earn this badge, the AEC must judge that the artifacts conform to the expectations set by the paper in terms of functionality, usability, and relevance. The AEC will consider four aspects of the artifacts in particular.
    • Documentation: are the artifacts sufficiently documented to enable them to be exercised by readers of the paper?
    • Completeness: do the submitted artifacts include all of the key components described in the paper?
    • Exercisability: do the submitted artifacts include the scripts and data needed to run the experiments described in the paper, and can the software be successfully executed?
    • Reusability: means that the artifacts are not just functional but of sufficient quality that they could be extended and reused by others.
  3. 🏆 IACR CHES Results Reproduced: To earn this badge, the AEC must judge that they can use the submitted artifacts to obtain the main results presented in the paper. In short, is it possible for the AEC to independently repeat the experiments and obtain results that support the main claims made by the paper? The goal of this effort is not to reproduce the results exactly but instead to generate results independently within an allowed tolerance such that the main claims of the paper are validated.

For more information, please see the TCHES 2024 Call for Artifacts.