Report

Message from the USENIX Security ’23 Artifact Evaluation Co-Chairs

On behalf of USENIX, we want to welcome you to the artifact evaluation (AE) proceedings of the 32nd USENIX Security Symposium. Over a year ago, we started to work with everyone to run an artifact evaluation in conjunction with the main conference. We are proud of what our community has accomplished together. Learning from the last years, we increased the artifact evaluation committee to 127 members. We mainly recruited the members via a self-nomination process leading to PhD students, post-docs or early career researchers from around the world to apply. One significant change was the introduction of a publication chair, since the expected workload to validate the numerous artifact appendices would have been too large.

Artifact Evaluation in the security community is still relatively new and only occurs for the 4th time since 2020. We aligned the artifact evaluation with the paper submission cycles such that the artifact evaluation follows after the camera-ready deadline of each of the 3 main paper submission cycles. Authors may submit their artifacts for review and select which badges they would like to be evaluated against. Each evaluation cycle took 4-5 weeks to complete in which the evaluators worked with the author-provided artifacts and discussed issues with the authors in a single-blind manner. In each cycle, evaluators were assigned 1 to 2 artifacts to judge against the badge criterion.

After three AE cycles, we conclude with 140 artifacts receiving at least one of the three badges. As a result, out of 442 USENIX Security 2023 papers 31% went through artifact evaluation and successfully received a badge, a decrease by 13% from 2022. In total, 138 Artifacts Available, 120 Artifacts Functional and 96 Results Reproduced badges were awarded by the Artifact Evaluation Committee (AEC). The Artifacts Available badge ensures that the artifact is publicly available; the Artifacts Functional badge ensures that the artifact is complete, documented, and exercisable; the Results Reproduced ensures that the major claims of the paper have been reproduced by an evaluator independently using the author-provided artifacts.

We are tremendously grateful to the AEC. The evaluators spent countless hours improving and communicating with the authors leading to more than 2,750 comments in HotCRP and close to 350 reviews. Without this effort this proceedings of 140 artifact appendices wouldn’t have been possible.

To further promote artifact evaluation in the community and highlight great artifacts, the AEC selected 4 distinguished artifact awards and 5 distinguished artifact reviewer awards. The former evidence artifacts that should act as light houses to anyone seeking guidance on how to build a great artifact. The latter are to recognize the efforts of reviewers who went above and beyond to review and help authors improve their artifacts.

New this year were updated documentation for authors and reviewers as well as the artifact appendix template. In addition, we allowed out-of-cycle submissions of artifacts. That is, authors were able to submit artifacts in the same cycle their paper was accepted or any of the following cycles. This resulted in a slow start to the artifact submissions in the summer cycle and doubled with every cycle leading to almost 80 artifact submissions in the final winter cycle.

We want to express our immeasurable gratitude to the community without whom these proceedings would not be possible.

Cristiano Giuffrida, VUSec Anjo Vahldiek-Oberwagner, Intel Labs USENIX Security ’23 Artifact Evaluation Co-Chairs

Alexios Voulimeneas, Delft University of Technology USENIX Security ‘23 Artifact Evaluation Publication chair