Stateful testing

Stateful testing introduces state machines.

Class-based state machines

class hegel.StateMachine

Base class for class-based state machines.

A state machine test checks a system through sequences of operations: each test case runs a sequence of rules chosen by the engine, checking the invariants along the way, and a failure is shrunk to a short sequence of steps. Rules and invariants are methods decorated with rule() and invariant(); each takes the test case for its step. The class attributes steps, min_concurrency and max_concurrency configure the run, as the Machine methods of the same names do; with max_concurrency > 1 the rules run on worker threads against the shared instance. A fresh instance is made for every test case: __init__ takes either no arguments or just the test case, to draw the initial state from.

class Counter(hegel.StateMachine):
    def __init__(self):
        self.count = 0

    @hegel.rule()
    def incr(self, tc):
        n = tc.draw(gs.integers(min_value=1, max_value=10))
        self.count += n

    @hegel.invariant()
    def non_negative(self, tc):
        assert self.count >= 0

test_counter = hegel.state_machine_test(Counter)
build_machine(tc: TestCase) → Machine

The Machine this instance runs as, with its rules, invariants and class settings.

hegel.rule(fn: Callable[[...], Any] | None = None, *, groups: Iterable[str] | None = None, weight: float = 1.0, name: str | None = None) → Any

Mark a StateMachine method as a rule: one step the engine may choose to run. The method takes the test case and draws whatever the step needs:

@hegel.rule()
def push(self, tc):
    self.stack.append(tc.draw(gs.integers()))

Usable as @hegel.rule or @hegel.rule(...). name replaces the method name in the failure report; groups and weight are as for Machine.rule(). A rule’s precondition is a TestCase.assume at its top.

hegel.invariant(fn: Callable[[...], Any] | None = None, *, always_run: bool = False, pause_faults: bool = False, name: str | None = None) → Any

Mark a StateMachine method as an invariant: a check of the machine’s state that fails by raising. The method takes the test case: def valid(self, tc).

Every invariant runs on the initial and final state. Hegel samples invariants at intermediate steps. Set always_run=True to check the invariant after every step. With pause_faults=True, fault injection is paused for the duration of each check: inside Antithesis the fuzzer stops injecting faults while the invariant runs; elsewhere the flag has no effect. Usable as @hegel.invariant or @hegel.invariant(...); name replaces the method name in the failure report.

hegel.state_machine_test(machine: type[StateMachine], **settings: Any) → Callable[[...], None]

A test function that runs machine, a StateMachine subclass, for pytest to collect when assigned to a test_ name:

test_counter = hegel.state_machine_test(Counter, test_cases=200)

Keyword arguments are settings, as for test(). A failure report says how to replay the failure by wrapping this call in reproduce_failure().

hegel.run_state_machine(tc: TestCase, machine: type[StateMachine] | StateMachine | Machine, **overrides: Any) → None

Run a state machine on tc: a Machine, a StateMachine instance, or a class to instantiate (with tc if its constructor accepts one). steps, min_concurrency and max_concurrency may be overridden; any other keyword is an InvalidArgument. Use it to run a machine from inside a test(), after other setup:

@hegel.test
def test_counter(tc):
    hegel.run_state_machine(tc, Counter, steps=100)

Function-based state machines

class hegel.Machine(tc: TestCase)

A state machine assembled from rule and invariant callables, inside a test: the function-based counterpart of StateMachine.

@hegel.test
def test_stack(tc):
    stack = []
    machine = hegel.Machine(tc)
    machine.rule("push", lambda tc: stack.append(tc.draw(gs.integers())))

    @machine.rule("pop")
    def pop(tc):
        tc.assume(stack)
        stack.pop()

    @machine.invariant("bounded")
    def bounded(tc):
        assert len(stack) <= 50

    machine.run()

Each rule and invariant takes the test case for its step, and draws from it. steps(), min_concurrency() and max_concurrency() return the machine, so calls chain.

With max_concurrency above 1 the machine runs concurrently, in rounds, on max_concurrency worker threads. The engine draws how many of them run rules in [min_concurrency, max_concurrency], weighted toward the maximum, and may redraw it between rounds. Each round, it picks one group of rules, every active worker runs a short sequence of that group’s rules against the shared model, and the invariants then run on the main thread. Rules in the same group may overlap; rules in different groups never do; a rule in several groups runs in the rounds of each of them, so a read rule in both a write and a flush group can overlap either. A concurrent run’s failures may not reproduce exactly, so they are confirmed by replay and reported with a caveat; see NondeterminismStrictness.

rule(name: str, apply: Callable[[TestCase], Any] | None = None, *, groups: Iterable[str] | None = None, weight: float = 1.0) → Any

Add a rule called name that runs apply(tc) as one step; usable as a decorator when apply is omitted.

A rule that rejects (with TestCase.assume) stops early without failing; rules should reject before they change the model. groups names the rule’s concurrency groups: rules in the same group may overlap, rules in different groups never do, and a rule in several groups runs in the rounds of each of them (rules without one share the anonymous group). weight is a positive hint of how often to choose the rule relative to the others. A duplicate name, a weight that is not finite and positive, or a single string in place of the list of groups is an InvalidArgument; so are an empty groups and one that repeats a group, which the engine reports when the machine runs.

invariant(name: str, apply: Callable[[TestCase], Any] | None = None, *, always_run: bool = False, pause_faults: bool = False) → Any

Add an invariant called name, checked by apply(tc); usable as a decorator when apply is omitted.

Every invariant runs on the initial and final state. Hegel samples invariants at intermediate steps. Set always_run to check the invariant after every step. With pause_faults set, fault injection is paused for the duration of each check: inside Antithesis the fuzzer stops injecting faults while the invariant runs, so it observes the system under test without a fault landing in the middle of the check; elsewhere the flag has no effect. A duplicate name is an InvalidArgument.

steps(step_count: int) → Machine

Set the number of steps per test case (default 50): every test case runs at least one and at most step_count, most run them all, and shrinking shortens a failing one. A step is one completed rule (one that rejects does not count) or, for a concurrent machine, one round. Returns the machine.

min_concurrency(n: int) → Machine

Set the fewest worker threads a concurrent run uses (default 1). Returns the machine.

max_concurrency(n: int) → Machine

Set the most worker threads a run uses (default 1, sequential). Above 1 the machine runs concurrently. Returns the machine.

run() → None

Run the machine on this test case: invariants on the initial state, then steps chosen by the engine, then invariants on the final state. The failure report lists each step’s rule and draws. A machine with no rules, or with min_concurrency above max_concurrency, is an InvalidArgument.

Pools

class hegel.Pool(tc: TestCase)

A pool of values a stateful test accumulates, with generators that draw from it: values_reusable() leaves the value in the pool, values_consumed() removes it. Drawing from an empty pool rejects the test case (so the rule stops early).

The engine tracks which value is drawn, so shrinking keeps the values a failure depends on and drops the rest; it shrinks toward values added earlier. Create the pool from the test case, typically in the machine’s __init__:

class Files(hegel.StateMachine):
    def __init__(self, tc):
        self.handles = hegel.Pool(tc)

    @hegel.rule()
    def open(self, tc):
        self.handles.add(open_file(tc.draw(gs.text(min_size=1))))

    @hegel.rule()
    def close(self, tc):
        tc.draw(self.handles.values_consumed()).close()

len(pool) is the number of values in it.

property is_empty: bool

Whether the pool has no values.

add(value: Any, tc: TestCase | None = None) → None

Add value to the pool. tc is the test case of the step adding it, by default the one the pool was created with.

values_reusable() → Generator[Any]

A generator of the pool’s values that leaves the drawn value in the pool.

values_consumed() → Generator[Any]

A generator of the pool’s values that removes the drawn value from the pool, for values a step uses up (a handle it closes, say). The value is removed as soon as it is drawn, even if the step then rejects. Filtering it removes only the value that passes.

class hegel.ConcurrentPool(tc: TestCase)

A Pool safe to share between the worker threads of a concurrent state machine. add() must be given the worker’s test case (the one the rule received), or it is an InvalidArgument.

property is_empty: bool

Whether the pool has no values.

add(value: Any, tc: TestCase | None = None) → None

Add value to the pool. tc is the test case of the step adding it, by default the one the pool was created with.