Testing Smart Contracts: Which Technique Performs Best?

Sefa Akca, Chao Peng, Ajitha Rajan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Background: Executing, verifying and enforcing credible transactions on permissionless blockchains is done using smart contracts. A key challenge with smart contracts is ensuring their correctness and security. Several test input generation techniques for detecting vulnerabilities in smart contracts have been proposed in the last few years. However, a comparison of proposed techniques to gauge their effectiveness is missing. Aim: This paper conducts an empirical evaluation of testing techniques for smart contracts. The testing techniques we evaluated are: (1) Blackbox fuzzing, (2) Adaptive fuzzing, (3) Coverage-guided fuzzing with an SMT solver and (4) Genetic algorithm. We do not consider static analysis tools, as several recent studies have assessed and compared effectiveness of these tools. Method: We evaluate effectiveness of the test generation techniques using (1) Coverage achieved - we use four code coverage metrics targeting smart contracts, (2) Fault finding ability - using artificially seeded and real security vulnerabilities of different types. We used two datasets in our evaluation - one with 1665 real smart contracts from Etherscan, and another with 90 real contracts with known vulnerabilities to assess fault finding ability. Result: We find Adaptive fuzzing performs best in terms of coverage and fault finding over contracts in both datasets. Conclusion: However, we believe considering dependencies between functions and handling Solidity specific features will help improve the performance of all techniques considerably.
Original languageEnglish
Title of host publicationProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)
Place of PublicationNew York, NY, USA
PublisherAssociation for Computing Machinery, Inc
Number of pages11
ISBN (Electronic)9781450386654
DOIs
Publication statusPublished - 11 Oct 2021
Event15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2021 - Online
Duration: 11 Oct 202115 Oct 2021
Conference number: 15
https://conf.researchr.org/home/esem-2021

Publication series

NameESEM '21
PublisherAssociation for Computing Machinery

Symposium

Symposium15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2021
Abbreviated titleESEM 2021
Period11/10/2115/10/21
Internet address

Keywords / Materials (for Non-textual outputs)

  • Input Generation
  • Fault Seeding
  • Blockchain
  • Fuzzer
  • Smart Contract
  • Constraint Solver
  • Genetic Algorithm
  • Ethereum

Fingerprint

Dive into the research topics of 'Testing Smart Contracts: Which Technique Performs Best?'. Together they form a unique fingerprint.

Cite this