Ontology-based method produces consistent, comparable security metrics for compliance checking
The claim
The proposed methodology systematically elicits security metrics that are consistent and cover standards comprehensively
Published as: “Developing Security Metrics for Automated Compliance Checking”
So what
The methodology could streamline the creation and sharing of security metrics, reducing redundancy and facilitating automated compliance across multiple standards.
The five-minute version
The authors show that an ontology-based elicitation method can systematically prevent duplicate metrics, ensure comparable abstraction levels, and support collaborative metric development via an open-source repository. They present a data model and a hybrid semi-structured/Rego description that enables automated compliance checking for standards such as EUCS and C5. The paper provides no empirical evaluation, so the strength of the evidence rests on the presented design and illustrative examples.
What they found 3
- Ontology-based elicitation prevents duplicate metrics and ensures comparable abstraction levels
“The ontology-based elicitation method systematically prevents the problems identified in Section 1. By requiring each metric to reference ontology concepts, the approach ensures: (1) no duplicate metrics are created for the same property, (2) all metrics operate at comparable abstraction levels defined by the ontology structure, (3) gaps in requirement coverage become visible through ontology analysis (identifying ontology properties which are not referenced in any metric), and (4) metrics from different sources can be integrated through shared ontological vocabulary.”
- The method enables automated compliance checking for modern standards
“The proposed metrics enable automated compliance checking, which is increasingly required by modern standards.”
- Metrics are described using a hybrid semi-structured format plus Rego implementation
“We adopt a hybrid approach that addresses the needs of diverse stakeholders. The representation employs a semi-structured format that maintains human readability while mandating the integration of concepts from our ontological basis.”
What undercuts it the paper’s own results
- First, automated compliance checking is limited by Goodhart’s law: metrics may cease to effectively measure security when used as key performance indicators, as auditees may optimize toward fulfilling metrics rather than implementing comprehensive security measures.
- Second, security cannot be fully captured by any finite set of metrics-they remain approximations of complex security properties.
- automatically checking multiple metrics cannot fully encompass the scope of prose-written requirements intended for human interpretation.
Stated limits 3
- First, automated compliance checking is limited by Goodhart’s law: metrics may cease to effectively measure security when used as key performance indicators, as auditees may optimize toward fulfilling metrics rather than implementing comprehensive security measures.
- Second, security cannot be fully captured by any finite set of metrics-they remain approximations of complex security properties.
- automatically checking multiple metrics cannot fully encompass the scope of prose-written requirements intended for human interpretation.
What’s new here 4
- Systematic ontology-based method for eliciting security metrics from standards
- Unified metric data model that links requirements to ontology concepts
- Hybrid representation combining human-readable semi-structured format with executable Rego policies
- Open-source collaborative repository (Cybersecurity Certification Hub) for shared metric development
Artifacts & methods 4
- ontology-based metric elicitation methodprimitive
- security metric data model (Table 1)theory
- hybrid semi-structured + Rego metric representationsystem
- Cybersecurity Certification Hub (open-source repository)system