Tests and test cases¶
Defining tests¶
- hegel.test(fn: Callable[[...], Any] | None = None, /, *, settings: Settings | None = None, **overrides: Any) Any¶
Decorate a test function taking a
TestCaseas its first argument (afterself, for a method) so that calling it, as pytest does, runs the property test.@hegel.test def test_reversing_twice(tc): xs = tc.draw(gs.lists(gs.integers())) assert list(reversed(list(reversed(xs)))) == xs @hegel.test(test_cases=500, derandomize=True) def test_with_settings(tc): ...
The decorated function takes the remaining parameters, which are passed through, so pytest fixtures and
parametrizework. Settings may be given assettings=(aSettings), as individual fields, or both, the fields overriding; withoutsettings=they start from the default profile, resolved each time the test runs. Failing examples are saved in the example database under the test’s module and qualified name, and replayed first on the next run.The test runs synchronously: an
async deftest, a class, or a function with no positional parameter for the test case is anInvalidArgument, as is calling a Hegel test from inside another. The wrapper hasis_hegel_test = Trueand the original function ashegel_inner_test.
- hegel.explicit_test_case(*args: Any, **kwargs: Any) Callable[[T], T]¶
Add an explicit test case: the test runs with these values before any are generated, in the
explicitphase. Positional values are consumed by draws in order; keyword values by draw name (x=1forx = tc.draw(...)).@hegel.explicit_test_case(xs=[]) @hegel.explicit_test_case(xs=[1, 1]) @hegel.test def test_sort_keeps_length(tc): xs = tc.draw(gs.lists(gs.integers())) assert len(sorted(xs)) == len(xs)
Stacked decorators run their cases top to bottom. Explicit cases are not shrunk, and their values are not checked against the generators: each draw simply returns the given value. Every value must be drawn, and every draw needs a value; a mismatch, a rejected assumption, or
draw_silent()in an explicit case is anInvalidArgument. The decorator may go above or belowtest().
- hegel.reproduce_failure(blob: str) Callable[[T], T]¶
Replay the failure that
blobencodes instead of generating test cases.A failure report ends with a line such as
@hegel.reproduce_failure("AXicY2Vg...")whenprint_blobis on; adding that decorator to atest()makes the next run consist only of replaying exactly that failing test case, which is useful when the failure is hard to find again. The decorators of several failures (from a report ofMultipleFailures, say) may be stacked: each blob is replayed, top to bottom, and every failure they reproduce is raised, together. If none of them fails any more, the run raisesDidNotReproduce. Remove the decorators once the bugs are fixed. An empty or non-string blob is anInvalidArgument.
- hegel.run(test_fn: Callable[[TestCase], Any], settings: Settings | None = None, *, database_key: str | None = None, test_location: TestLocation | None = None, explicit_cases: Iterator[ExplicitCase] | list[ExplicitCase] | tuple[ExplicitCase, ...] = (), reproduce_blob: str | Iterable[str] | None = None, reproducer: Callable[[str], str] = default_reproducer, **overrides: Any) None¶
Run
test_fnas a property test, without a decorator: it is called once per test case with aTestCase, and the first exception it raises (other than a rejected assumption) is a failure, reported after shrinking: the drawn values, any caveat, and how to reproduce it are added to the exception as notes, and the exception is re-raised. A passing run returnsNone.settings(withoverridesapplied, as forSettings) configures the run; by default it isSettings().database_keyidentifies the test in the example database (without one, failures are not saved or replayed) andtest_locationin the engine’s reports.explicit_casesrun first, with their given values.reproduce_blob(one blob, or a collection of them) makes the run consist only of replaying previously reported failures, each blob in turn: every failure they reproduce is raised, together, and when none reproduces the run raisesDidNotReproduce.reproducerwords the note saying how to replay a failure, given its blob.def check(tc): xs = tc.draw(gs.lists(gs.integers())) assert sorted(sorted(xs)) == sorted(xs) hegel.run(check, test_cases=1000)
test()is this, run when the decorated function is called.
- hegel.main(fn: Callable[[TestCase], Any] | None = None, /, *, settings: Settings | None = None, **overrides: Any) Any¶
Make a function that takes a
TestCasethe entry point of a program, such as an Antithesis workload. Calling the decorated function reads the command line and runs one test case.@hegel.main def workload(tc): n = tc.draw(gs.integers()) assert n + 0 == n if __name__ == "__main__": workload()
A failure is raised as from
test(), and its report ends with arerun with: --failure-blob BLOBline. Thetoo_slowandtest_cases_too_largehealth checks are always suppressed.Each setting except
test_casesis a command-line option named after itshegel.tomlkey, with-in place of_. Values use the same format as inhegel.toml, and options override the decorator’s settings.test_casescannot be set.--profile NAMEuses the named settings profile in place of the default one, and the decorator’s keyword settings apply on top of it. It does not change asettings=object.--failure-blob BLOBreplays a failure.--helplists the options.An invalid command line argument prints a usage message and exits with status 2.
- hegel.composite(function: Callable[[...], T]) Callable[[...], Generator[T]]¶
Define a generator as a function taking a
TestCasefirst.Calling the decorated function with the remaining arguments returns a generator; each draw runs the function body with the test case and those arguments, and the generated value is what the body returns. The body draws with
tc.drawlike a test does, so later draws can depend on earlier ones. Its own draws are not printed separately: the failure report prints the composite’s value.@hegel.composite def sorted_pairs(tc, limit): a = tc.draw(gs.integers(0, limit)) b = tc.draw(gs.integers(a, limit)) return (a, b) @hegel.test def test_pairs(tc): a, b = tc.draw(sorted_pairs(100)) assert a <= b
Decorating a class, or a function without a positional first parameter for the test case, is an
InvalidArgument.hegel.compositeandhegel.generators.compositeare the same function.
The test case¶
- class hegel.TestCase¶
One execution of a property test: the object a test draws its inputs from, and through which it rejects inputs, adds notes to the failure report, steers generation and records statistics.
A test decorated with
hegel.test()receives aTestCaseas its first argument, once per test case; tests never construct one. An instance belongs to the test case it was passed for and must not be kept and used after the test returns. It may be driven by one thread at a time: to draw from several threads, give each thread its ownclone().@hegel.test def test_addition_commutes(tc): x = tc.draw(gs.integers()) y = tc.draw(gs.integers()) tc.note(f"x + y = {x + y}") assert x + y == y + x
- draw(generator: Generator[T], name: str | None = None) T¶
Draw a value from
generatorand report it.Passing something other than a
Generator(a generator function that was not called, say) is anInvalidArgument.The value is printed, as
name = value, into the output shown for a failing test case. Whennameis omitted it is inferred from the assignment the call appears in (x = tc.draw(...)); a draw that can run more than once per test case (in a loop, say) is numbered, asx_1,x_2, … An explicitnameis always numbered, since it typically names a draw in a helper function, and an unassigned draw reports asdraw_1,draw_2, … Names are inferred from the test’s source, so draws in code whose source is unavailable report asdraw_N.The generator prints the value as it draws it, so it must be printable: every built-in generator is, and so is a
Generatorsubclass that implementsdo_draw_and_print. A subclass that implements onlydo_draw(or a structure containing one) raisesNotPrintableon the first test case that draws it. Implementdo_draw_and_print, make it printable withprint_with()orprint_as_repr(), or draw it withdraw_silent().Draws made inside a
composite()generator, or while another draw is in progress, are never printed on their own, not even when the outer draw raises: they are part of the outer value, and a draw that raises prints nothing at all. They still need a printable generator.
- draw_silent(generator: Generator[T]) T¶
Draw a value from
generatorwithout printing it in the failure report. Any generator can be drawn silently, printable or not. Use it for values that would only clutter the report, or in a customGenerator’sdo_draw.
- assume(condition: object) None¶
Reject the current test case unless
conditionis true.A rejected test case is neither a pass nor a failure: it is abandoned and does not count toward
test_cases. Use it for inputs the property does not apply to:x = tc.draw(gs.integers()) y = tc.draw(gs.integers()) tc.assume(y != 0) assert (x // y) * y + x % y == x
Rejecting most test cases makes a run slow and fails the
filter_too_muchhealth check; a generator that only produces valid inputs is better where one is easy to write.
- reject() None¶
Reject the current test case unconditionally, as
tc.assume(False)does. It never returns: it raisesAssumeFailed, which Hegel catches.
- note(message: str) None¶
Add
messageto the output of this test case, between the drawn values in the order it was recorded.The output is shown for the failing test case Hegel reports, and for every test case at
verbosity="verbose"and above; otherwise notes cost nothing. Use it for intermediate values that help explain a failure:result = compute(x) tc.note(f"intermediate result: {result!r}")
- target(score: float, label: str = '') None¶
Guide generation toward test cases that maximise
score.scoreis anintorfloatandlabelnames the quantity being maximised, so one test can target several. After a warm-up, the engine’s targeting phase searches around the highest-scoring test cases for higher scores. This helps find failures that need an extreme value of some measure (a long queue, a deep tree). Each label can be targeted at most once per test case; a second call with the same label, or a score or label of the wrong type, is anInvalidArgument.queue_length = simulate(tc) tc.target(queue_length, label="queue length")
- event(label: str) None¶
Record that
labelhappened in this test case.With
show_statistics=True(orHEGEL_STATISTICS=1), the end of the run reports the percentage of test cases recording each event, which shows whether the inputs exercise what they should:tc.event("empty" if not xs else "non-empty")
- event_value(label: str, value: float) None¶
Record a numeric observation
valueunderlabel.With
show_statistics=Truethe end of the run reports the count, minimum, median, mean, 90th percentile and maximum of the values recorded under each label.valuemust be anintorfloat, or it is anInvalidArgument.tc.event_value("list length", len(xs))
- repeat(body: Callable[[], Any]) NoReturn¶
Run
bodyan engine-chosen number of times, then end the test case successfully.bodytakes no arguments and draws from this test case (a closure overtc). A repetition that rejects (withassume()) is skipped and the next one runs. A failure in any repetition fails the test case, and shrinking reduces the number of repetitions along with the values each drew. Each repetition is reported under a# Repetition #Nheading. In an explicit test case,bodyruns once.repeatnever returns normally: it raises an internal control-flow signal that ends the test body, so code after it does not run. It suits workload-style tests that perform a sequence of operations until the engine decides to stop:@hegel.test def test_store(tc): store = Store() def step(): key = tc.draw(gs.text()) store.put(key, tc.draw(gs.integers())) assert store.get(key) is not None tc.repeat(step)
- clone() TestCase¶
Another
TestCasefor the same test case, to draw from on another thread.A
TestCasemay be driven by one thread at a time, so a test that draws from several threads gives each its own clone. A clone draws from a choice stream of its own, so the threads’ draws do not perturb each other and the whole test case still shrinks and replays. Its drawn values and notes appear in the failure report in a section of their own, placed where the clone was made, and it shares draw names with this one (a draw namedxon each prints asx_1andx_2). Everything else, such as the test case’s outcome, is shared.A draw on a clone can end the test case by raising, like any other draw, so a thread should hand its exception back to the test to re-raise rather than let it escape the thread:
@hegel.test def test_list_append_is_thread_safe(tc): shared = [] clones = [tc.clone() for _ in range(4)] errors = [] def worker(i): try: clones[i].set_worker(i) for _ in range(clones[i].draw(gs.integers(0, 100), name="n")): shared.append(i) except BaseException as error: errors.append(error) threads = [threading.Thread(target=worker, args=(i,)) for i in range(4)] for thread in threads: thread.start() for thread in threads: thread.join() if errors: raise errors[0] assert len(shared) == sum(shared.count(i) for i in range(4))
- set_worker(index: int) None¶
Attribute this
TestCase’s output to workerindex: every line it prints from now on, drawn values and notes alike, is prefixed with[worker N +X.XXXms], the time since the test case started, so a report from several threads shows how their lines interleaved. Clones made from it afterwards inherit the attribution.indexis a non-negative integer, or it is anInvalidArgument.
Settings¶
Settings describes how settings are resolved, the profiles, and hegel.toml.
- class hegel.Settings(parent: Settings | None = None, **overrides: Any)¶
Configuration for a run. Immutable.
A
Settingsvalue holds every field.Settings(**overrides)resolves the default profile when it is created and keeps a copy of the result, with the given fields set over it: thedefaultset withset_default_profile(),HEGEL_DEFAULT_PROFILE, thedefaultentry inhegel.toml, or the environment:workloadunder Antithesis,cion a CI server,developmentotherwise. TheHEGEL_*environment variables (HEGEL_TEST_CASES,HEGEL_SEED, …) apply over the profile, and the given fields over both. A malformedhegel.tomlor variable, or an unknown default profile, is anInvalidArgumentraised here. The copy does not change afterwards: aSettingsmade before the default profile changes, at import time in aconftest.pysay, keeps the values it was made with, so make it afterregister_profile()andset_default_profile(), or givehegel.test()its fields as keywords, which it resolves each time the test runs.Settings(parent, **overrides)copiesparent’s fields and setsoverridesover them.Usually settings are given to
hegel.test()as keywords (@hegel.test(test_cases=500)), which override the same fields.The fields:
test_casesHow many test cases to run (default 100). Rejected test cases do not count.
verbosityHow much Hegel prints: a
Verbosity(defaultnormal).seedAn integer seed for a reproducible run, or
None(the default) for a fresh random seed each run.derandomizeDerive a fixed seed from the test, so every run of it is the same (default
False; theciprofile sets it). Aseedtakes precedence.databaseWhere failing examples are saved and replayed from:
Nonefor the engine’s default directory (.hegel/examples), a directory path, or""(orFalse) to disable the database. Note thatNonedoes not disable it, unlike in the Rust library and Hypothesis.suppress_health_checkThe
HealthCheckmembers not to fail the run on (default none).phasesThe
Phasemembers to run (default all).report_multiple_failuresReport every distinct failure found, as a
MultipleFailures, rather than only the smallest (defaultFalse).stop_on_first_failureStop generating once a failure is found (default
True). WhenFalse, the run uses its wholetest_casesbudget before reporting; every failure found is saved to the database, and withreport_multiple_failuresoff only the smallest is shrunk and reported.show_statisticsPrint the end-of-run statistics of the events recorded with
event()andevent_value()(defaultFalse).unbounded_choicesLet a test case make any number of choices (default
False). Otherwise a test case that makes more than 2**20 choices (draws, and the steps of collections and stateful runs) is stopped and discarded. A long-running test case, such as a concurrent state machine driven for hours, needs the limit removed; the cost is the memory to record every choice. SuppressingHealthCheck.test_cases_too_largeremoves the limit too.print_blobOn failure, say how to replay it with
reproduce_failure()(defaultTrue).backendThe source of randomness: a
Backend(defaultdefault).nondeterminism_strictnessHow to react to a test that behaves differently on replay: a
NondeterminismStrictness(defaultquiet).
Enum-valued fields take members (
HealthCheck.filter_too_much) or their names ("filter_too_much"); the set-valued fields take one or a collection of them. A value of the wrong type is anInvalidArgument, and an unknown field aTypeError.thorough = hegel.Settings(test_cases=10_000, suppress_health_check=["too_slow"]) @hegel.test(settings=thorough) def test_something(tc): ...
- classmethod from_profile(name: str, **overrides: Any) Settings¶
Settings resolved, now, from the named profile rather than the default one, with
overridesapplied. The profile is one of the engine’s (development,ci,workload, orbasefor the plain defaults), one defined inhegel.toml, or one registered withregister_profile(); an unknown name is an error. UnlikeSettings(), this ignores which profile is the default.
- static register_profile(name: str, settings: Settings | None = None, **overrides: Any) None¶
Register (or replace) a named profile, for the whole process:
settings(the default profile when omitted) withoverridesapplied, resolved now. The profile can then be selected withfrom_profile(),set_default_profile(),HEGEL_DEFAULT_PROFILEor pytest’s--hegel-profile.hegel.Settings.register_profile("nightly", test_cases=10_000) hegel.Settings.set_default_profile("nightly")
Registering a profile is not retroactive: a
Settingsvalue already made keeps its fields, so register profiles before making the settings that should start from them. The reserved namesbaseanddefaultcannot be registered.
- static set_default_profile(name: str | None) None¶
Make the named (already registered) profile the one
Settings()resolves, for the whole process.Nonerestores the engine’s own choice (seeSettings). An unknown profile name is an error. Not retroactive: aSettingsvalue already made keeps its fields.
- class hegel.Verbosity(*values)¶
How much output Hegel produces during a run.
quiet: nothing; a failing test still raises, without the drawn values attached.normal: the default. A failing test’s drawn values and notes, attached to the exception, and any warnings.verbose: also every test case’s drawn values and notes as it runs, a failing test case’s exception on one line (where it was raised, its type and the first line of its message), and the start and end of each phase.debug: also a failing test case’s full traceback, in place of that line, and the engine’s own diagnostics.
Settings accept a member or its name:
verbosity="verbose".- quiet = 1¶
- normal = 0¶
- verbose = 2¶
- debug = 3¶
- class hegel.Phase(*values)¶
The phases of a run, in the order they run. By default all of them do;
phasesselects a subset.explicit: run the explicit test cases added withexplicit_test_case().reuse: replay failures saved in the example database.generate: generate new test cases.target: steer generation toward hightarget()scores.shrink: shrink a failing test case to a minimal one. Without it, the first failure found is reported as it is.
Phase.all()is the set of every phase.- explicit = 1¶
- reuse = 2¶
- generate = 4¶
- target = 8¶
- shrink = 16¶
- class hegel.HealthCheck(*values)¶
Health checks: proactive errors raised when a test is likely to test poorly or too slowly. A failing check raises
FailedHealthCheck;suppress_health_checkturns checks off for a test whose behaviour is intentional.filter_too_much: too many test cases are rejected, byassume()orfilter().too_slow: the first test cases take too long to run: more than 30 seconds in all before ten valid ones, counting the test body and its generators.test_cases_too_large: test cases routinely reach the limit on the number of choices one may make. Suppressing it also removes that limit, as theunbounded_choicessetting does.large_initial_test_case: the smallest possible test case is already very large.
HealthCheck.all()is the set of every check.- filter_too_much = 1¶
- too_slow = 2¶
- test_cases_too_large = 4¶
- large_initial_test_case = 8¶
- classmethod all() frozenset[HealthCheck]¶
- class hegel.Backend(*values)¶
Where the engine’s randomness comes from.
default: a seeded pseudo-random generator, so runs are reproducible fromseedand shrinking and replay work as usual.urandom: fresh entropy from outside the process on every draw, for running under Antithesis, whose fuzzer then controls every choice. Theworkloadprofile selects it.
- default = 1¶
- urandom = 2¶
- class hegel.NondeterminismStrictness(*values)¶
How a run reacts to a test whose behaviour changes when the same choices are replayed (a test depending on timing, threads, or state left over from an earlier test case).
quiet: the default. Switch silently to confirming each failure by repeated replay; the report carries a caveat describing how reliably it reproduced.warn: the same, with a one-line notice when the switch happens.error: abort the run withFlaky, as Hypothesis does.
- quiet = 0¶
- warn = 1¶
- error = 2¶
Explicit test cases and locations¶
- class hegel.ExplicitCase(args: tuple[Any, ...], kwargs: dict[str, Any])¶
The values of one explicit test case, as declared with
@hegel.explicit_test_case(...): positional values are consumed by draws in order, keyword values by the draw’s name.
- class hegel.TestLocation(file: str, begin_line: int, class_name: str = '', function: str = '')¶
Where a test is defined: the file, first line, class and function name.
hegel.test()records it automatically;hegel.run()takes one to identify the test in the engine’s reports.
Versions¶
- hegel.__version__¶
str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to ‘utf-8’. errors defaults to ‘strict’.