Embedding and classifying test execution traces using neural networks

Foivos Tsimpourlas, Gwenyth Rooijackers, Ajitha Rajan, Miltiadis Allamanis

Research output: Contribution to journalArticlepeer-review


Classifying test executions automatically as pass or fail remains a key challenge in software testing and is referred to as the test oracle problem. It is being attempted to solve this problem with supervised learning over test execution traces. A programme is instrumented to gather execution traces as sequences of method invocations. A small fraction of the programme's execution traces is labelled with pass or fail verdicts. Execution traces are then embedded as fixed length vectors and a neural network (NN) component that uses the line-by-line information to classify traces as pass or fail is designed. The classification accuracy of this approach is evaluated using subject programs from different application domains—1. Module from Ethereum Blockchain, 2. Module from PyTorch deep learning framework, 3. Microsoft SEAL encryption library components, 4. Sed stream editor, 5. Nine network protocols from Linux packet identifier, L7-Filter and 6. Utilities library, commons-lang for Java. For all subject programs, it was found that test execution classification had high precision, recall and specificity, averaging to 93%, 94% and 96%, respectively, while only training with an average 14% of the total traces. Experiments show that the proposed NN-based approach is promising in classifying test executions from different application domains.

Original languageEnglish
Pages (from-to)301-316
Number of pages16
JournalIET Software
Issue number3
Early online date17 Aug 2021
Publication statusPublished - 1 Jun 2022


  • execution trace
  • neural networks
  • software testing
  • test oracle


Dive into the research topics of 'Embedding and classifying test execution traces using neural networks'. Together they form a unique fingerprint.

Cite this