Nondeterministic tests

Hegel assumes that a test is deterministic: run it twice with the same choices and it does the same thing. Shrinking relies on this. The engine changes a failing input a little, reruns the test, and checks whether it still fails. If the test can fail or pass on the same input, shrinking cannot tell whether a change made a difference.

Some tests aren’t deterministic: they depend on timing, threads, the current time or an external service. Concurrent state machines are nondeterministic by nature. Hegel notices when replaying the same choices gives a different result, and handles it according to the nondeterminism_strictness setting.

Quiet handling (the default)

With nondeterminism_strictness="quiet", Hegel switches to handling the test as nondeterministic. It confirms each failure by replaying it several times, and reports it with a caveat saying how reliably it reproduced. A failure that reproduced only once is dropped if another failure is confirmed.

This is the right setting for tests that are nondeterministic on purpose.

Warnings and errors

nondeterminism_strictness="warn" handles the test the same way, but prints a one-line notice when Hegel detects the nondeterminism, so you find out about it.

nondeterminism_strictness="error" treats nondeterminism as a bug in the test: the run stops with a hegel.Flaky error. This is what Hypothesis does.

Nondeterministic generation

A different kind of nondeterminism is a test that makes different requests of the engine when given the same choices: drawing from a different generator, or a different number of times, depending on something other than the values it has drawn. Hegel cannot replay such a test at all, and reports it as an error whatever the setting.