Umbra

Coverage says what ran. It cannot say what was missed.

Every other code-quality tool reports what it found. Coverage says which lines ran. A linter says which lines smell. A green test suite says the tests it has all pass. None of them report what the tests are structurally unable to see, and that blind region, the umbra, is exactly where generated code fails in production.

This is the Slop Audit's argument at the scale of one function: coverage measures what ran, not all the ways the code could run. You can hit 100% coverage and still have no warning that any given input will break you, because coverage only ever sees the paths your tests already walk. Every instrument has a blind spot it cannot see by construction, and failure lives in that blind spot.

The Silence index

Umbra reports a Silence index: the honest complement to coverage. Where coverage is the percentage of lines that ran, the Silence index is the percentage of closeable facets that the suite leaves silent, the reachable, meaningful cases nothing asserts. It is shown beside the coverage number it corrects. Higher means more blind spots.

Silence index = 100 × closeable silence sites ÷ total closeable facets

A facet is one branch, one candidate input region, one annotated return contract for a called function, one reachable exception path, or one runtime honesty property. It is closeable when Umbra has enough evidence to decide whether the suite closes it. The index is not a vibe: it comes from a body of formal software-measurement work, the same premise the Slop Audit rests on, given an honest denominator.

What Umbra does

For a module and its test suite, Umbra does four things:

The output is a report made for humans, not the usual wall of test-runner text: a short account that ends on a real, failing test Umbra wrote, sitting in the blind spot your green suite swore did not exist. It renders in the terminal, as Markdown, and as a self-contained HTML page.

def total(items: list[float], discount: float) -> float:
    return sum(items) - discount

# Umbra's generated proof — verified fail
args = ([10.0], -2.0)
result = total(*args)
assert result == 10.0    # a negative discount should not increase the total
# AssertionError: 12.0 == 10.0

Six languages

Umbra audits Python, TypeScript, JavaScript, Java, Rust, and Go. Each language is parsed with tree-sitter, then Umbra reads the test file to identify its framework and runs the matching one, taking coverage from that language's own tool. A framework it cannot identify, or a run it cannot complete, is reported as not measured with the reason attached, never a guessed number.

Rust is the showcase. A Rust enum is a finite Set with exhaustive match, exactly the case Honest Test calls total: a bounded space you can enumerate rather than sample. Auditing Rust, Umbra treats each enum variant as its own region and reports the variants a test never constructs. Bounded types get enumerated, not guessed.

Silence you close with a type, not a test

When a function declares no types, plain JavaScript, or Python with its annotations stripped, Umbra has no map of valid inputs, so it refuses to fake a low score. It reports 100% silence, with the plain reason that the domain is undeclared, and names the remedy as declaring a type, not writing a test. You cannot show a test suite covers a domain the author never stated.

Coverage-proven branches still get real failing-test proofs even there, because coverage measures them directly without a type. So the two kinds of silence stay distinct: ordinary silence you close by writing a test, and undeclared-domain silence you close by declaring a type.

Two extensions

Harden mode turns the audit into a loop. It generates the tests that close each open facet, keeps the passing ones as regression guards and the failing ones as bug proofs, re-audits, and repeats until no closeable silence remains. The Silence index is the loop's convergence target, so "done" is definable and measurable, unlike coverage-chasing, which never ends and can be gamed.

The call-stack map classifies each function's operations into four roles, input boundary, orchestrator, pure core, output boundary, and renders them with I/O on the outside and pure logic in the middle. A write sitting in the pure lane is an honesty violation, visible as a shape. Umbra emits the same map as a plain-text architecture spec in the Honest Framework's FOSS .hd format, so the audit is interoperable, not just a picture.

Used where structure runs out, and nowhere else

The core design decision is where not to use the model. The deterministic layer, reachability and branch analysis, which invariants the types encode, what the suite exercises, runs with zero model calls. Only once a silence is located mechanically does GPT-5.6 come in, and it receives one already-isolated function, its type contract, and one gap description, never the repository, never the test files, never any other function.

Every case the model proposes is executed. Only cases that actually fail survive into the report. The model suggests; the runtime adjudicates through an execution gate. No claim ships that the tool cannot reproduce. It is the same discipline Umbra applies to your code, turned on its own tools: name what an instrument can and cannot judge, and use it only where it can.

Made to run inside the agent's loop

Umbra's home is the agent's own loop. An agent writes hundreds of lines in a minute, and nobody hand-runs an audit on every generation, so the missing case slips through precisely because, at that moment, no one is there to think of it. Umbra runs where the code is born: as a plugin inside the coding agent, it audits what was just written, names the silence, and hands back the failing test, in the same loop, before you read the diff. It also runs as a standalone command-line tool for CI and local use.

It audits itself

The strongest thing you can say about an honesty tool is that it survives its own audit. Point Umbra at its own core and it reports 94.4% coverage and 65.4% Silence index on its largest module, 138 closeable silences across 211 facets: near-total line coverage, and two-thirds of the meaningful facets still unasserted. It does not flinch, and it does not pretend the number is zero.

And it goes past the number. Run on itself, it caught a real bug in its own verification gate: generated proofs were loading the audited module anonymously, so any code with relative imports died at the import before the assertion ever ran. It found the hole in itself, and the regression test is the receipt. The auditor passed its own audit by first failing it.

Lineage

Umbra is not invented from nothing. The Slop Audit applies one premise, every instrument reveals only what it was built to reveal and is silent on the rest, to an enterprise codebase and its compliance posture, across 18 dimensions and four layers. Umbra applies the same premise to a single module and its test suite. Same idea, smaller target.

The Silence index rests on a second Foundation standard, Honest Test, which makes the coverage argument precise: unit tests and property tests both sample from a potentially infinite input space, so they can never reach total coverage, while a bounded space can be exhausted and made total. Honest Test is the constructive answer, shrink the space until you can run all of it. Umbra is the diagnostic complement for the code you cannot shrink: it measures the gap between what your sampling touched and what total coverage would require, and reports that gap as the Silence index.

Get it

Umbra is open source under Apache-2.0. Run it against your own modules, in any of the six languages, and cite it, without asking permission and without buying anything.

• Umbra on GitHub (source, the Codex plugin, and the reproducible benchmark)
• The Slop Audit, the standard Umbra is the module-scale instrument of
• The Honest Framework in context at the Open Honest Foundation
• [email protected]