Tests and test cases

Defining tests

hegel.test(fn: Callable[[...], Any] | None = None, /, *, settings: Settings | None = None, **overrides: Any) → Any

Decorate a test function taking a TestCase as its first argument (after self, for a method) so that calling it, as pytest does, runs the property test.

@hegel.test
def test_reversing_twice(tc):
    xs = tc.draw(gs.lists(gs.integers()))
    assert list(reversed(list(reversed(xs)))) == xs

@hegel.test(test_cases=500, derandomize=True)
def test_with_settings(tc): ...

The decorated function takes the remaining parameters, which are passed through, so pytest fixtures and parametrize work. Settings may be given as settings= (a Settings), as individual fields, or both, the fields overriding; without settings= they start from the default profile, resolved each time the test runs. Failing examples are saved in the example database under the test’s module and qualified name, and replayed first on the next run.

The test runs synchronously: an async def test, a class, or a function with no positional parameter for the test case is an InvalidArgument, as is calling a Hegel test from inside another. The wrapper has is_hegel_test = True and the original function as hegel_inner_test.

hegel.explicit_test_case(*args: Any, **kwargs: Any) → Callable[[T], T]

Add an explicit test case: the test runs with these values before any are generated, in the explicit phase. Positional values are consumed by draws in order; keyword values by draw name (x=1 for x = tc.draw(...)).

@hegel.explicit_test_case(xs=[])
@hegel.explicit_test_case(xs=[1, 1])
@hegel.test
def test_sort_keeps_length(tc):
    xs = tc.draw(gs.lists(gs.integers()))
    assert len(sorted(xs)) == len(xs)

Stacked decorators run their cases top to bottom. Explicit cases are not shrunk, and their values are not checked against the generators: each draw simply returns the given value. Every value must be drawn, and every draw needs a value; a mismatch, a rejected assumption, or draw_silent() in an explicit case is an InvalidArgument. The decorator may go above or below test().

hegel.reproduce_failure(blob: str) → Callable[[T], T]

Replay the failure that blob encodes instead of generating test cases.

A failure report ends with a line such as @hegel.reproduce_failure("AXicY2Vg...") when print_blob is on; adding that decorator to a test() makes the next run consist only of replaying exactly that failing test case, which is useful when the failure is hard to find again. The decorators of several failures (from a report of MultipleFailures, say) may be stacked: each blob is replayed, top to bottom, and every failure they reproduce is raised, together. If none of them fails any more, the run raises DidNotReproduce. Remove the decorators once the bugs are fixed. An empty or non-string blob is an InvalidArgument.

hegel.run(test_fn: Callable[[TestCase], Any], settings: Settings | None = None, *, database_key: str | None = None, test_location: TestLocation | None = None, explicit_cases: Iterator[ExplicitCase] | list[ExplicitCase] | tuple[ExplicitCase, ...] = (), reproduce_blob: str | Iterable[str] | None = None, reproducer: Callable[[str], str] = default_reproducer, **overrides: Any) → None

Run test_fn as a property test, without a decorator: it is called once per test case with a TestCase, and the first exception it raises (other than a rejected assumption) is a failure, reported after shrinking: the drawn values, any caveat, and how to reproduce it are added to the exception as notes, and the exception is re-raised. A passing run returns None.

settings (with overrides applied, as for Settings) configures the run; by default it is Settings(). database_key identifies the test in the example database (without one, failures are not saved or replayed) and test_location in the engine’s reports. explicit_cases run first, with their given values. reproduce_blob (one blob, or a collection of them) makes the run consist only of replaying previously reported failures, each blob in turn: every failure they reproduce is raised, together, and when none reproduces the run raises DidNotReproduce. reproducer words the note saying how to replay a failure, given its blob.

def check(tc):
    xs = tc.draw(gs.lists(gs.integers()))
    assert sorted(sorted(xs)) == sorted(xs)

hegel.run(check, test_cases=1000)

test() is this, run when the decorated function is called.

hegel.main(fn: Callable[[TestCase], Any] | None = None, /, *, settings: Settings | None = None, **overrides: Any) → Any

Make a function that takes a TestCase the entry point of a program, such as an Antithesis workload. Calling the decorated function reads the command line and runs one test case.

@hegel.main
def workload(tc):
    n = tc.draw(gs.integers())
    assert n + 0 == n

if __name__ == "__main__":
    workload()

A failure is raised as from test(), and its report ends with a rerun with: --failure-blob BLOB line. The too_slow and test_cases_too_large health checks are always suppressed.

Each setting except test_cases is a command-line option named after its hegel.toml key, with - in place of _. Values use the same format as in hegel.toml, and options override the decorator’s settings. test_cases cannot be set.

--profile NAME uses the named settings profile in place of the default one, and the decorator’s keyword settings apply on top of it. It does not change a settings= object. --failure-blob BLOB replays a failure. --help lists the options.

An invalid command line argument prints a usage message and exits with status 2.

hegel.composite(function: Callable[[...], T]) → Callable[[...], Generator[T]]

Define a generator as a function taking a TestCase first.

Calling the decorated function with the remaining arguments returns a generator; each draw runs the function body with the test case and those arguments, and the generated value is what the body returns. The body draws with tc.draw like a test does, so later draws can depend on earlier ones. Its own draws are not printed separately: the failure report prints the composite’s value.

@hegel.composite
def sorted_pairs(tc, limit):
    a = tc.draw(gs.integers(0, limit))
    b = tc.draw(gs.integers(a, limit))
    return (a, b)

@hegel.test
def test_pairs(tc):
    a, b = tc.draw(sorted_pairs(100))
    assert a <= b

Decorating a class, or a function without a positional first parameter for the test case, is an InvalidArgument. hegel.composite and hegel.generators.composite are the same function.

The test case

class hegel.TestCase

One execution of a property test: the object a test draws its inputs from, and through which it rejects inputs, adds notes to the failure report, steers generation and records statistics.

A test decorated with hegel.test() receives a TestCase as its first argument, once per test case; tests never construct one. An instance belongs to the test case it was passed for and must not be kept and used after the test returns. It may be driven by one thread at a time: to draw from several threads, give each thread its own clone().

@hegel.test
def test_addition_commutes(tc):
    x = tc.draw(gs.integers())
    y = tc.draw(gs.integers())
    tc.note(f"x + y = {x + y}")
    assert x + y == y + x
draw(generator: Generator[T], name: str | None = None) → T

Draw a value from generator and report it.

Passing something other than a Generator (a generator function that was not called, say) is an InvalidArgument.

The value is printed, as name = value, into the output shown for a failing test case. When name is omitted it is inferred from the assignment the call appears in (x = tc.draw(...)); a draw that can run more than once per test case (in a loop, say) is numbered, as x_1, x_2, … An explicit name is always numbered, since it typically names a draw in a helper function, and an unassigned draw reports as draw_1, draw_2, … Names are inferred from the test’s source, so draws in code whose source is unavailable report as draw_N.

The generator prints the value as it draws it, so it must be printable: every built-in generator is, and so is a Generator subclass that implements do_draw_and_print. A subclass that implements only do_draw (or a structure containing one) raises NotPrintable on the first test case that draws it. Implement do_draw_and_print, make it printable with print_with() or print_as_repr(), or draw it with draw_silent().

Draws made inside a composite() generator, or while another draw is in progress, are never printed on their own, not even when the outer draw raises: they are part of the outer value, and a draw that raises prints nothing at all. They still need a printable generator.

draw_silent(generator: Generator[T]) → T

Draw a value from generator without printing it in the failure report. Any generator can be drawn silently, printable or not. Use it for values that would only clutter the report, or in a custom Generator’s do_draw.

assume(condition: object) → None

Reject the current test case unless condition is true.

A rejected test case is neither a pass nor a failure: it is abandoned and does not count toward test_cases. Use it for inputs the property does not apply to:

x = tc.draw(gs.integers())
y = tc.draw(gs.integers())
tc.assume(y != 0)
assert (x // y) * y + x % y == x

Rejecting most test cases makes a run slow and fails the filter_too_much health check; a generator that only produces valid inputs is better where one is easy to write.

reject() → None

Reject the current test case unconditionally, as tc.assume(False) does. It never returns: it raises AssumeFailed, which Hegel catches.

note(message: str) → None

Add message to the output of this test case, between the drawn values in the order it was recorded.

The output is shown for the failing test case Hegel reports, and for every test case at verbosity="verbose" and above; otherwise notes cost nothing. Use it for intermediate values that help explain a failure:

result = compute(x)
tc.note(f"intermediate result: {result!r}")
target(score: float, label: str = '') → None

Guide generation toward test cases that maximise score.

score is an int or float and label names the quantity being maximised, so one test can target several. After a warm-up, the engine’s targeting phase searches around the highest-scoring test cases for higher scores. This helps find failures that need an extreme value of some measure (a long queue, a deep tree). Each label can be targeted at most once per test case; a second call with the same label, or a score or label of the wrong type, is an InvalidArgument.

queue_length = simulate(tc)
tc.target(queue_length, label="queue length")
event(label: str) → None

Record that label happened in this test case.

With show_statistics=True (or HEGEL_STATISTICS=1), the end of the run reports the percentage of test cases recording each event, which shows whether the inputs exercise what they should:

tc.event("empty" if not xs else "non-empty")
event_value(label: str, value: float) → None

Record a numeric observation value under label.

With show_statistics=True the end of the run reports the count, minimum, median, mean, 90th percentile and maximum of the values recorded under each label. value must be an int or float, or it is an InvalidArgument.

tc.event_value("list length", len(xs))
repeat(body: Callable[[], Any]) → NoReturn

Run body an engine-chosen number of times, then end the test case successfully.

body takes no arguments and draws from this test case (a closure over tc). A repetition that rejects (with assume()) is skipped and the next one runs. A failure in any repetition fails the test case, and shrinking reduces the number of repetitions along with the values each drew. Each repetition is reported under a # Repetition #N heading. In an explicit test case, body runs once.

repeat never returns normally: it raises an internal control-flow signal that ends the test body, so code after it does not run. It suits workload-style tests that perform a sequence of operations until the engine decides to stop:

@hegel.test
def test_store(tc):
    store = Store()

    def step():
        key = tc.draw(gs.text())
        store.put(key, tc.draw(gs.integers()))
        assert store.get(key) is not None

    tc.repeat(step)
clone() → TestCase

Another TestCase for the same test case, to draw from on another thread.

A TestCase may be driven by one thread at a time, so a test that draws from several threads gives each its own clone. A clone draws from a choice stream of its own, so the threads’ draws do not perturb each other and the whole test case still shrinks and replays. Its drawn values and notes appear in the failure report in a section of their own, placed where the clone was made, and it shares draw names with this one (a draw named x on each prints as x_1 and x_2). Everything else, such as the test case’s outcome, is shared.

A draw on a clone can end the test case by raising, like any other draw, so a thread should hand its exception back to the test to re-raise rather than let it escape the thread:

@hegel.test
def test_list_append_is_thread_safe(tc):
    shared = []
    clones = [tc.clone() for _ in range(4)]
    errors = []

    def worker(i):
        try:
            clones[i].set_worker(i)
            for _ in range(clones[i].draw(gs.integers(0, 100), name="n")):
                shared.append(i)
        except BaseException as error:
            errors.append(error)

    threads = [threading.Thread(target=worker, args=(i,)) for i in range(4)]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()
    if errors:
        raise errors[0]
    assert len(shared) == sum(shared.count(i) for i in range(4))
set_worker(index: int) → None

Attribute this TestCase’s output to worker index: every line it prints from now on, drawn values and notes alike, is prefixed with [worker N +X.XXXms], the time since the test case started, so a report from several threads shows how their lines interleaved. Clones made from it afterwards inherit the attribution. index is a non-negative integer, or it is an InvalidArgument.

Settings

Settings describes how settings are resolved, the profiles, and hegel.toml.

class hegel.Settings(parent: Settings | None = None, **overrides: Any)

Configuration for a run. Immutable.

A Settings value holds every field. Settings(**overrides) resolves the default profile when it is created and keeps a copy of the result, with the given fields set over it: the default set with set_default_profile(), HEGEL_DEFAULT_PROFILE, the default entry in hegel.toml, or the environment: workload under Antithesis, ci on a CI server, development otherwise. The HEGEL_* environment variables (HEGEL_TEST_CASES, HEGEL_SEED, …) apply over the profile, and the given fields over both. A malformed hegel.toml or variable, or an unknown default profile, is an InvalidArgument raised here. The copy does not change afterwards: a Settings made before the default profile changes, at import time in a conftest.py say, keeps the values it was made with, so make it after register_profile() and set_default_profile(), or give hegel.test() its fields as keywords, which it resolves each time the test runs. Settings(parent, **overrides) copies parent’s fields and sets overrides over them.

Usually settings are given to hegel.test() as keywords (@hegel.test(test_cases=500)), which override the same fields.

The fields:

test_cases

How many test cases to run (default 100). Rejected test cases do not count.

verbosity

How much Hegel prints: a Verbosity (default normal).

seed

An integer seed for a reproducible run, or None (the default) for a fresh random seed each run.

derandomize

Derive a fixed seed from the test, so every run of it is the same (default False; the ci profile sets it). A seed takes precedence.

database

Where failing examples are saved and replayed from: None for the engine’s default directory (.hegel/examples), a directory path, or "" (or False) to disable the database. Note that None does not disable it, unlike in the Rust library and Hypothesis.

suppress_health_check

The HealthCheck members not to fail the run on (default none).

phases

The Phase members to run (default all).

report_multiple_failures

Report every distinct failure found, as a MultipleFailures, rather than only the smallest (default False).

stop_on_first_failure

Stop generating once a failure is found (default True). When False, the run uses its whole test_cases budget before reporting; every failure found is saved to the database, and with report_multiple_failures off only the smallest is shrunk and reported.

show_statistics

Print the end-of-run statistics of the events recorded with event() and event_value() (default False).

unbounded_choices

Let a test case make any number of choices (default False). Otherwise a test case that makes more than 2**20 choices (draws, and the steps of collections and stateful runs) is stopped and discarded. A long-running test case, such as a concurrent state machine driven for hours, needs the limit removed; the cost is the memory to record every choice. Suppressing HealthCheck.test_cases_too_large removes the limit too.

print_blob

On failure, say how to replay it with reproduce_failure() (default True).

backend

The source of randomness: a Backend (default default).

nondeterminism_strictness

How to react to a test that behaves differently on replay: a NondeterminismStrictness (default quiet).

Enum-valued fields take members (HealthCheck.filter_too_much) or their names ("filter_too_much"); the set-valued fields take one or a collection of them. A value of the wrong type is an InvalidArgument, and an unknown field a TypeError.

thorough = hegel.Settings(test_cases=10_000, suppress_health_check=["too_slow"])

@hegel.test(settings=thorough)
def test_something(tc): ...
classmethod from_profile(name: str, **overrides: Any) → Settings

Settings resolved, now, from the named profile rather than the default one, with overrides applied. The profile is one of the engine’s (development, ci, workload, or base for the plain defaults), one defined in hegel.toml, or one registered with register_profile(); an unknown name is an error. Unlike Settings(), this ignores which profile is the default.

static register_profile(name: str, settings: Settings | None = None, **overrides: Any) → None

Register (or replace) a named profile, for the whole process: settings (the default profile when omitted) with overrides applied, resolved now. The profile can then be selected with from_profile(), set_default_profile(), HEGEL_DEFAULT_PROFILE or pytest’s --hegel-profile.

hegel.Settings.register_profile("nightly", test_cases=10_000)
hegel.Settings.set_default_profile("nightly")

Registering a profile is not retroactive: a Settings value already made keeps its fields, so register profiles before making the settings that should start from them. The reserved names base and default cannot be registered.

static set_default_profile(name: str | None) → None

Make the named (already registered) profile the one Settings() resolves, for the whole process. None restores the engine’s own choice (see Settings). An unknown profile name is an error. Not retroactive: a Settings value already made keeps its fields.

replace(**overrides: Any) → Settings

A copy of these settings with overrides applied.

class hegel.Verbosity(*values)

How much output Hegel produces during a run.

  • quiet: nothing; a failing test still raises, without the drawn values attached.

  • normal: the default. A failing test’s drawn values and notes, attached to the exception, and any warnings.

  • verbose: also every test case’s drawn values and notes as it runs, a failing test case’s exception on one line (where it was raised, its type and the first line of its message), and the start and end of each phase.

  • debug: also a failing test case’s full traceback, in place of that line, and the engine’s own diagnostics.

Settings accept a member or its name: verbosity="verbose".

quiet = 1
normal = 0
verbose = 2
debug = 3
class hegel.Phase(*values)

The phases of a run, in the order they run. By default all of them do; phases selects a subset.

  • explicit: run the explicit test cases added with explicit_test_case().

  • reuse: replay failures saved in the example database.

  • generate: generate new test cases.

  • target: steer generation toward high target() scores.

  • shrink: shrink a failing test case to a minimal one. Without it, the first failure found is reported as it is.

Phase.all() is the set of every phase.

explicit = 1
reuse = 2
generate = 4
target = 8
shrink = 16
classmethod all() → frozenset[Phase]
class hegel.HealthCheck(*values)

Health checks: proactive errors raised when a test is likely to test poorly or too slowly. A failing check raises FailedHealthCheck; suppress_health_check turns checks off for a test whose behaviour is intentional.

  • filter_too_much: too many test cases are rejected, by assume() or filter().

  • too_slow: the first test cases take too long to run: more than 30 seconds in all before ten valid ones, counting the test body and its generators.

  • test_cases_too_large: test cases routinely reach the limit on the number of choices one may make. Suppressing it also removes that limit, as the unbounded_choices setting does.

  • large_initial_test_case: the smallest possible test case is already very large.

HealthCheck.all() is the set of every check.

filter_too_much = 1
too_slow = 2
test_cases_too_large = 4
large_initial_test_case = 8
classmethod all() → frozenset[HealthCheck]
class hegel.Backend(*values)

Where the engine’s randomness comes from.

  • default: a seeded pseudo-random generator, so runs are reproducible from seed and shrinking and replay work as usual.

  • urandom: fresh entropy from outside the process on every draw, for running under Antithesis, whose fuzzer then controls every choice. The workload profile selects it.

default = 1
urandom = 2
class hegel.NondeterminismStrictness(*values)

How a run reacts to a test whose behaviour changes when the same choices are replayed (a test depending on timing, threads, or state left over from an earlier test case).

  • quiet: the default. Switch silently to confirming each failure by repeated replay; the report carries a caveat describing how reliably it reproduced.

  • warn: the same, with a one-line notice when the switch happens.

  • error: abort the run with Flaky, as Hypothesis does.

quiet = 0
warn = 1
error = 2

Explicit test cases and locations

class hegel.ExplicitCase(args: tuple[Any, ...], kwargs: dict[str, Any])

The values of one explicit test case, as declared with @hegel.explicit_test_case(...): positional values are consumed by draws in order, keyword values by the draw’s name.

class hegel.TestLocation(file: str, begin_line: int, class_name: str = '', function: str = '')

Where a test is defined: the file, first line, class and function name. hegel.test() records it automatically; hegel.run() takes one to identify the test in the engine’s reports.

Versions

hegel.__version__

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to ‘utf-8’. errors defaults to ‘strict’.

hegel.engine_version() → str

The version of the libhegel engine linked into this build.