The Test Oracle Problem

The test oracle needs to address several questions:

  • What is the source for the expected result?
    • is the source trusted and correct?
    • a correct specification isn't always available
  • Is it possible to define the expected result precisely?
    • is there a precision requirement?
    • there are different kinds of input with different possible precision
  • What is the format and media of the expected result?
    • is it displayed? printed? streamed into a file? database? network?
  • What should remain unchanged by the test input?
  • Is it feasible to directly or indirectly compare the actual result with the expected result?

Specified Test Oracles

Created from a trusted (partial) specification or abstract model.

  • trusted specification: precondition/postcondition
  • abstract model: finite state machine

Sometimes implemented as built-in assertions in the source code. A failure occurs when an assertion evaluates to false.

Implicit Test Oracles

Distinguishes between a system's correct and incorrect behavior through general implicit knowledge rather than domain knowledge or specification.

Implicit test oracles target faults by demonstrating runtime anomalies such as exceptions, abnormal termination, and crashes. This applies to almost all programs.

Derived Test Oracles

Distinguishes correct behavior based on information derived from properties of the software under test, other increments, or implementations of it.

Regression testing can exploit relations that should hold across multiple versions of the software.