Probing and Fault Injection of Dependable Distributed Protocols

Ensuring that a distributed system with strict dependability constraints meets its prescribed specification is a growing challenge that confronts software developers and system engineers. This paper presents a technique for probing and fault injection of fault-tolerant distributed protocols. The proposed technique, called script-driven probing and fault injection, can be used for studying the behaviour of distributed systems and for detecting design and implementation errors of fault-tolerant protocols. The focus of this work is on fault injection techniques that can be used to demonstrate three aspects of a target protocol : detection of design or implementation errors ; identification of violations of protocol specifications ; and insight into design decisions made by the implementers. The emphasis of our approach is on experimental techniques intended to identify specific problems in a protocol or its implementation rather than the evaluation of system dependability through statistical metrics such as fault coverage. To demonstrate the capabilities of this technique, the paper describes a probing and fault injection tool, called the PFI tool (probe/fault injection tool), and a summary of several extensive experiments that studied the behaviour of two protocols : the Transmission Control Protocol (TCP) and a group membership protocol (GMP).