The Slop Audit
Code coverage cannot prove every case was tested.
If your organization has to attest to the correctness of its software — regulated industries, banks, insurers, pharmaceutical and medical-device manufacturers, public infrastructure — you are almost certainly leaning on a number that does not mean what you were told it means: test coverage. The Slop Audit is an open standard that measures what coverage cannot: whether a codebase is even structurally capable of having every behavior verified. It is stewarded by the Open Honest Foundation, released under Apache-2.0 and is the common name for the standard formally designated the Honest Audit.
Run the audit on your own code. Paste a public GitHub repository and read the Layer 1 panel. Nothing is installed, and your code is never executed.
Coverage is not proof of correctness
The conventional standard for software testing is line coverage, or coverage for short. Coverage tells you how many lines of your code was executed by tests.
Importantly, it does not tell you how much of your software's actual behavior was properly verified. You can hit 95% coverage and still leave the vast majority of real-world scenarios, the ones regulators and auditors care about, completely untested.
Whether true verification is even possible depends on the structure of the code itself. The way most code is written leaves the door open for many different combinations of steps and logic: each one of these is a "state permutation." When code has lots of state permutation, the number of possible behaviors explodes far beyond what any test suite can cover. More tests don't solve this.
We measured it. We analyzed 200 established, community-recognized production applications (50 each in Python, Java, TypeScript and C#), every one with at least 100 GitHub stars, at least 10,000 lines of production code, a year or more of active history and a real test suite. These were not random repositories: the corpus is weighted toward blue-chip enterprise code, published by financial-services firms, big-tech consumer platforms and major enterprise-software vendors. This is the code the industry holds up as its best work.
Even there, roughly 99% of typical enterprise systems (outside of React front-ends) turned out to be structurally incapable of being fully verified through testing because of this state explosion. The Slop Audit measures this risk directly through objective mathematical calculations, which give you real evidence of correctness instead of misleading coverage percentages.
The finite-testability indicators
State permutation is not a vibe; it is measured. Three Layer 1 indicators quantify whether a codebase is structurally capable of exhaustive verification at all, before any question of how many tests exist. They are the mathematical core of the audit.
Two of these three need your test suite to run, so the hosted version at try.slopaudit.org cannot report them in full: it gives the decision count for L1.19 without the exercised fraction, and it does not report L1.20 at all. It answers whether a codebase can be exhaustively verified, which is a property of how it is built. Whether it is verified is a property of the test suite, and answering that means running the tool locally. A repository with no tests at all can score well on the hosted card.
L1.18, mutable-state ratio. The percentage of functions whose behavior depends on state outside their parameter list. A pure function is determined entirely by its inputs; a method that reads instance, inherited or global state depends on the order of every prior call, and its effective state space multiplies with each mutable reference. The higher the ratio, the larger the portion of the codebase that is mathematically untestable regardless of test budget. This is the flagship measurement, and it is empirically grounded: across 200 public repositories (50 each in Python, Java, TypeScript, C#), median mutable-state ratios were Python 61.1%, Java 53.0%, C# 40.0% and TypeScript 14.8%, with TypeScript bimodal: a React frontend subpopulation below 15% against a backend subpopulation of 40 to 65% (Wasserman 2026, Finite Testability of Enterprise Software).
L1.19, decision-space coverage. The honest version of test coverage. Of the decisions that are finitely enumerable — dispatch-table keys, match/case arms, enum branches, configuration-flag values — what percentage are exercised by at least one test? Line coverage can read 85% while decision-space coverage is 40%: most lines run, but only a fraction of the actual decisions the code can make have ever been verified. Standard coverage tools do not report this; it requires counting decision points and matching them against test invocations.
L1.20, test determinism. Do the tests produce the same result regardless of execution order? Pure-function tests are order-independent by construction; tests that lean on mutable fixtures, singletons or class-level setup often pass or fail depending on order, which means the suite is not a reliable verification instrument in the first place. Measured by running the full suite in randomized order five times and counting how many runs pass cleanly.
Together these three answer the question coverage cannot: is exhaustive behavioral verification even possible for this codebase? Where the mutable-state ratio is high, the decision space is under-exercised or the tests are order-dependent, the answer is no, and no volume of additional tests changes it. The remaining seventeen Layer 1 indicators (delete/add ratio, fuzzy duplication, god-file concentration, type-escape density, secrets scan and the rest) surround this core; the finite-testability triad is what makes the Slop Audit a measure of structural verifiability rather than a gauge of how AI-generated code looks.
You cannot eyeball quality
We all know what AI Slop is. The word gets used to describe a general impression: code that looks AI-generated, rushed or low-quality. However, a polished codebase can hide serious defects, and rough-looking code can be as solid as can be. The Slop Audit replaces a general impression with a measurement: an evidence-based, reproducible score against named indicators and compliance thresholds, created from hard data. Only the actual structure decides.
The audit scores any codebase against the published thresholds of named compliance frameworks across 18 dimensions covering security architecture, data architecture, compliance engineering, operational security, performance engineering, operations, DevOps, infrastructure, software architecture, governance, process engineering, lifecycle management and software development.
It applies to any codebase, in any language, regardless of architectural style. Each indicator is calibrated against the compliance-framework thresholds it maps to.
The four-layer model
The Slop Audit is not a single methodology. It is four nested methodologies that compose, each operating at a different cognitive level and each producing a different kind of evidence. The boundary between Layer 3 and Layer 4 is the load-bearing decision of the entire standard: Phase 0 runs Layers 1, 2 and 3, which trained assessors can reproduce; Phase 1 adds Layer 4, the elite architectural judgment that transfers only by apprenticeship.
| Layer | Name | Judgment | Phase |
|---|---|---|---|
| 1 | Quantitative git-history assessment | Mechanical, no judgment | Phase 0 |
| 2 | Quantitative artifact assessment | Mechanical, no judgment | Phase 0 |
| 3 | Qualitative specified judgment | Trainable, specified markers | Phase 0 |
| 4 | Architectural synthesis | Elite, by apprenticeship | Phase 1 |
Layer 1: git-history assessment
A mechanical analysis of the codebase git history and static artifacts that produces 20 numerical indicators: mutable-state ratio, decision-space coverage, test determinism, delete/add ratio, secrets scan, type-escape density, fuzzy duplication, "god-file" concentration and 12 others. Inputs are a git repository and a date range. The same indicators applied to the same repository by two different assessors produce identical numbers. Numbers that come from running git commands are harder to dispute than numbers that come from architectural judgment.
Layer 2: artifact assessment
A mechanical inspection of structural artifacts (configuration, dependency declarations, file structures, the presence or absence of specific kinds of files) for each of the 18 dimensions, scored Present, Partial, Absent or Not Applicable, with file paths and line numbers cited as evidence. Every finding can be reproduced by any other trained assessor in minutes. The audit software is not asked to trust judgment; it looks at the codebase's own files and confirms what is or is not there.
Layer 3: specified judgment
A structured exercise of qualitative judgment applied through specified markers, on aspects of each dimension that cannot be reduced to artifact inspection. Each dimension defines between three and five markers, each scored against defined criteria, with a stated inter-rater-reliability target. The judgment is real, but the criteria are explicit enough that trained assessors produce consistent scores. This is what lets Phase 0 answer the meaningful question on dimensions like pattern sophistication and architectural philosophy without requiring elite experience.
Layer 4: architectural synthesis
The judgment that cannot be reduced to specified markers: whether an architecture is the right one for the domain, whether hidden couplings will surface under operational stress, whether patterns are used correctly in the deepest sense. This requires years of building, shipping and maintaining systems at scale, and it does not transfer through documentation. During a Phase 0 audit, Layer 4 issues are recorded as flagged for Phase 1 follow-up; they are not scored and do not influence the Layer 2 and Layer 3 dimension scores.
A Phase 0 audit of a mid-size codebase (50,000 to 250,000 lines) runs in about five days and produces a Slop Report: the Layer 1 panel, the combined Layer 2 and Layer 3 scorecard for all 18 dimensions and the Layer 4 flagged-findings section. It also yields a SOC 2 deliverable: a compliance-evidence package a CIO can hand to their SOC 2 auditor as a byproduct of the Phase 0 audit.
The 18 dimensions
Each dimension is inspected mechanically at Layer 2 and, where applicable, assessed against specified markers at Layer 3. Each maps to the published thresholds of the compliance frameworks below.
Mapped to named compliance frameworks
The audit does not invent its own thresholds. Each indicator and dimension is calibrated against the published thresholds of established compliance and quality frameworks, so a score is auditable against standards a compliance officer already recognizes:
- SOC 2 Trust Services Criteria
- NIST SP 800-53
- OSFI B-13
- OWASP ASVS
- ISO/IEC 25010
- WCAG 2.2, Section 508, EN 301 549, and AODA (accessibility)
- Quebec Law 25
Research and status
The Slop Audit methodology and its underlying measurements are published under the Open Honest research program, pre-registered on the Open Science Framework and independently validated.
- OSF DBSYG block (construct and predictive validity studies): osf.io/dbsyg, DOI 10.17605/OSF.IO/DBSYG
- Independent validation: institutional collaboration of record at UQAM/LATECE; assessor protocols and inter-rater-reliability calibration in preparation
- Methodology: complete, approximately 45,000 words of documented procedures across 18 dimensions, four layers and the SOC 2 deliverable extraction
- The structural rules the audit measures against are the Honest Code Principles, twenty-three principles each naming a category of defect and removing it. The Honest Code Principles are one way to pass this audit by construction; other architectures pass too, and the audit is independent of any of them.
- Released under Apache-2.0; the full specification, the Layer 1 reference implementation and the conformance suite are published at github.com/openhonest/slop-audit
Run it
Layer 1 is mechanical, so it ships as a tool. The measurements on this page are the ones it produces; the same repository audited by two assessors returns the same numbers.
try.slopaudit.org
Paste a public repository and read the card. Nothing to install. Your code is cloned, measured statically and deleted.
The portable binary
One self-contained file per platform, with the grammars for nine languages linked in. No interpreter, no install and no network, for a client site or an air-gapped machine.
The reference implementation
The canonical Python instrument. It runs the target's own test suite for the two runtime indicators, which the hosted version cannot do.
Frequently asked questions
What is a Slop Audit?
The Slop Audit is an open standard that measures whether a codebase is structurally capable of exhaustive behavioral verification. It produces an evidence-based, reproducible score of any codebase, in any language, against the published thresholds of named compliance frameworks across 18 dimensions and four layers. It is stewarded by the Open Honest Foundation and released under Apache-2.0. Its formal designation is the Honest Audit.
Is the Slop Audit about detecting AI-generated code?
No. The Slop Audit measures structural verifiability, not authorship. It does not judge whether code looks machine-written or human-written, and it is not a detector of "AI slop." A polished codebase can hide serious defects, and rough-looking code can be structurally sound; only the actual structure decides. What the audit measures is whether a codebase's state-permutation space is small enough for its behavior to be exhaustively tested at all.
How is the Slop Audit different from code coverage?
Code coverage tells you how many lines of code a test suite executed. It cannot tell you whether every behavior was verified or whether exhaustive verification is even possible. The Slop Audit measures that directly, through finite-testability indicators: chiefly the mutable-state ratio (L1.18), decision-space coverage (L1.19) and test determinism (L1.20). A codebase can report 95% line coverage and still be structurally incapable of full verification because of state-space explosion.
What is the mutable-state ratio (L1.18)?
The mutable-state ratio is the percentage of a codebase's functions whose behavior depends on state outside their parameter list. A pure function is determined entirely by its inputs; a method that reads instance, inherited or global state depends on the order of every prior call, so its effective state space multiplies with each mutable reference. The higher the ratio, the larger the portion of the codebase that is mathematically untestable regardless of test budget. Across 200 public repositories, median ratios were Python 61.1%, Java 53.0%, C# 40.0% and TypeScript 14.8%.
What does the Slop Audit measure, and against what standards?
Four nested layers: Layer 1 produces twenty mechanical git-history indicators, including the finite-testability triad; Layer 2 is mechanical artifact inspection across all 18 dimensions; Layer 3 is specified qualitative judgment; Layer 4 is elite architectural synthesis. Scores are calibrated against the published thresholds of SOC 2, NIST SP 800-53, OSFI B-13, OWASP ASVS, ISO/IEC 25010 and accessibility standards, so a result is auditable against frameworks a compliance officer already recognizes.
Who created the Slop Audit, and is it free to use?
The Slop Audit was created by Adam Zachary Wasserman and is stewarded by the Open Honest Foundation. The specification, the 18-dimension catalog and the Layer 1 indicator definitions are published under Apache-2.0. You can adopt it, run it against your own codebases and cite it without asking permission and without buying anything.
Use it
The Slop Audit is an open standard. You can adopt it, run it against your own codebases and cite it, without asking permission and without buying anything. The specification, the 18-dimension catalog and the Layer 1 indicator definitions are published under Apache-2.0.
- The standard in context at the Open Honest Foundation
- The Slop Audit on GitHub (specification, Layer 1 reference implementation and conformance suite)
- [email protected]