Jussipekka Leiwo

— On the schedule —

IT Security Product Certification in the Era of Automated Evidence Generation and AI-Assisted Evaluation

The classic certification of an IT Security product followed the certify and freeze model, where the certification target is certified once and frozen. The certificate is assigned to a precise release of the product and remains valid until the end of the life time of the release - or when the certificate was revoked. Most certification standards include a paradigm for the maintenance of the certificates. Yet, the classic model has been challenged on each side of the Atlantic by means which go way beyond the intended use of the assurance maintenance: The EU Cyber Resiliency Act (CRA) mandates vulnerability monitoring, timely patching of the vulnerabilities, and a commitment to the maintenance of the certification target throughout it's life time. In the North America, the certificate life times have been relatively short, and a re-certification has been required to extend the certification target life time. NIAP mandates the ability to upgrade the software in most certification targets, and has moved to an active overseeing of the certificate validity through regular SBOM re-validation against databases of known vulnerabilities. Both of the above have lead to a continuous certification paradigm. Once a certification target has been certified, the developer is responsible of ensuring that new releases are produced in accordance with the certified life-cycle model of the certification target. There are additional trends which will substantially influence the certifications in the foreseeable future and challenge our idea of security assurance:Test automation using AI-based and non-AI based tools in the evaluation of the certification targets. There is a race going on in the test automation. In particular, in the evaluations against NIAP collaborative Protection Profiles (cPP) where the tests have been exactly presribed, the potential for the automation is substantial. It is foreseeable, that there are tests which cannot be automated. Yet, a good understanding of what can be automated will allow the evaluation facilities a major leap in productivity. The European Union Common Criteria (EUCC) evaluation paradigm is quite different. The tests are designed by the evaluator based on the design and implementation of the particular certification Target (i.e. Target of Evaluation, TOE) instead of being prescribed by the certification scheme owner. Yet, the potential for automation is equally high.AI-assisted production of the evaluation evidence. In particular in the Common Criteria certifications, the developer's documentation burden is considerable. The evaluator will also have to examine the evidence for conformance with the Common Evaluation Methodology (CEM) and device the penetration tests based on the documentation. It is foreseeable, that the developers will in the future use AI-assisted tools in the authoring of the evaluation evidence. Combining the above, there exists a potential - maybe even likely - future, where AI-based tools evaluate the evidence produced by other AI-based tools. While conceptually exciting, this is not futuristic. The development is going on right now, and the outcome will change the way we define assurance on IT Security Products. This presentation will explore the term assurance, and how our assurance paradigm will have to change in order to adapt the change. The presentation shall be pragmatic, yet explorative rather than definitive. There are no clear models for security assurance in the above future. Yet, this is a discussion that the industry must have in order to understand the trustworthiness of security products in the future. No large language models were harmed in the making of this proposal. None will be harmed in the presentation.