Evaluation

To help support the reproducibility for research results, IEEE S&P encourages authors of accepted papers to submit software they develop and datasets they use to perform their research and make them publicly available to the entire community. We believe that this is an important initiative that can help the entire community increase its reputation, and make research in the security field advance faster by taking advantage of systems previously built by other researchers.

We required all code submissions to be tested on publicly available research infrastructure, whenever possible. Our evaluation process leveraged public research infrastructure (SPHERE, CloudLab and Chameleon). More than half of the artifacts across both cycles were evaluated on SPHERE, with a few evaluated on CloudLab and Chameleon. We believe this helped make the badged research artifacts easily reusable.

We also standardized artifact metadata and used packaging code to encode the input from authors in a standardized, machine-readable format.

We thank all the authors and the evaluators who participated in this initiative!