Requirements Coverage as an Adequacy Measure for Conformance Testing

Ajitha Rajan, Michael Whalen, Matt Staats, Mats P.E. Heimdahl

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Conformance testing in model-based development refers to the testing activity that verifies whether the code generated (manually or automatically) from the model is behaviorally equivalent to the model. Presently the adequacy of conformance testing is inferred by measuring structural coverage achieved over the model. We hypothesize that adequacy metrics for conformance testing should consider structural coverage over the requirements either in place of or in addition to structural coverage over the model. Measuring structural coverage over the requirements gives a notion of how well the conformance tests exercise the required behavior of the system.

We conducted an experiment to investigate the hypothesis stating structural coverage over formal requirements is more effective than structural coverage over the model as an adequacy measure for conformance testing. We found that the hypothesis was rejected at 5% statistical significance on three of the four case examples in our experiment. Nevertheless, we found that the tests providing requirements coverage found several faults that remained undetected by tests providing model coverage. We thus formed a second hypothesis stating that complementing model coverage with requirements coverage will prove more effective as an adequacy measure than solely using model coverage for conformance testing. In our experiment, we found test suites providing both requirements coverage and model coverage to be more effective at finding faults than test suites providing model coverage alone, at 5% statistical significance. Based on our results, we believe existing adequacy measures for conformance testing that only consider model coverage can be strengthened by combining them with rigorous requirements coverage metrics.
Original languageEnglish
Title of host publicationFormal Methods and Software Engineering
EditorsShaoying Liu, Tom Maibaum, Keijiro Araki
PublisherSpringer-Verlag GmbH
Pages86-104
Number of pages19
Volume5256
ISBN (Electronic)978-3-540-88194-0
ISBN (Print)978-3-540-88193-3
DOIs
Publication statusPublished - 2008

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Berlin / Heidelberg
Volume5256
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Fingerprint Dive into the research topics of 'Requirements Coverage as an Adequacy Measure for Conformance Testing'. Together they form a unique fingerprint.

Cite this