The problem with agentic scanning
A plausible, well-evidenced, wrong finding is worse than no finding.
It burns the credibility of every other finding in the report. LLM agents are excellent at locating candidate issues across a large codebase and unreliable at adjudicating them — they will rate a public court record as a leaked secret, confidently, with a quote and a HIGH. So whoever has to act on the report ends up re-deriving it themselves, and the assessment has cost more than it saved.
"ssn": "078-05-1120", "dob": "1935-04-02"
A public court record, filed in a docket anyone can read, used as a test fixture. Not a secret. Never was. The retraction stays in the report, where the reader can see that someone checked.
Volume is not coverage
A scanner emits hundreds of candidates and ranks them by pattern confidence, not by whether the path is actually reachable. Nobody triages all of it, so nobody trusts any of it.
You cannot tell which findings were checked
A report that mixes verified issues with unconfirmed agent output, and labels neither, is not a verdict. It is a worklist wearing a verdict's clothes.
"What's wrong" without "since when"
A list of defects tells you nothing about whether the risk arrived in one bad fortnight or accreted over three years — which is the thing that decides what you do on Monday.
Codebase security & supportability assessment
Agents find. You adjudicate.
Assay gives one engineer the reach of nine parallel security lenses across an unfamiliar codebase — then puts every finding through a human verification gate before it earns a place in the report. Evidence-labelled. Reproducible. And your source never has to leave your infrastructure.
--offline runs air-gapped and makes provably zero network callsWho it's for
Built for security reviews you have to defend, and for technical due diligence.
Both need the same thing and rarely get it: an assessment of somebody else's codebase where each claim can be traced back to the line of source that supports it.
Security review & compliance
- Every high-severity finding confirmed against the primary source, and labelled as such
- A verification attestation — critical/high coverage, evidence tiers, retractions kept
- Findings export as SARIF 2.1.0 straight into GitHub code scanning or Microsoft Defender
- Runs inside your own network, so a code-egress clause is never the blocker
Technical due diligence
- Security, code quality and long-term supportability assessed in one pass
- Every finding pinned to the commit that introduced it — one bad quarter, or three years?
- Bus factor, churn hotspots, complexity and an AI-authorship signal, measured from git
- A weighted readiness score, so the verdict rolls into one number a deal team can act on
How a codebase assessment works
A method, not just a scanner.
Five stages. A depth flag scales the scope — --light, --thorough
or --full — but the gate in the middle is never skipped. That gate is what makes
the output worth reading.
Recon
Map the stack, domain, data sensitivity and which gates already exist — into one recon map every lens reads first.
Fan out
Nine lenses run concurrently, each narrowed to the surfaces recon actually found. Quote file:line, trace to sink, refute first.
Verify
Every high-severity finding re-checked by a human in the primary source. Non-negotiable at every depth.
Synthesize
An answer-first executive summary plus a labelled technical appendix — every finding carrying its evidence tier.
Publish
A self-verifying .docx, PDF, HTML or SARIF, plus the interactive history dashboard.
L6.3), never an ordinal, because
ordinals move the moment a render is narrowed to the repo in front of you. The prompts the
lenses run are generated from that registry and the build fails if the two drift, so adding a
check for an emergent threat is one record — and every report can say exactly which question
raised each finding.Before stage 00
Auditing a repository means opening it. That is itself the attack surface.
An assessment runs with the untrusted tree as the working directory of a session that holds shell and file-write authority. Part of that tree is never read by a model at all — it is consumed by the agent harness. A configuration loader has no reading. It applies the file.
| Channel | What the harness does with it |
|---|---|
CLAUDE.md, AGENTS.md |
Concatenated into the session's system prompt — before any skill or preamble the model reads |
.claude/settings.json |
Supplies permissions.allow and hook definitions, plus keys that each name a command the harness runs |
.claude/hooks/ |
Scripts a hook definition names; run on tool events |
.mcp.json |
Declares tool servers the session may call, and the commands that start them |
.claude/agents, commands, skills |
Define agents, commands and skills the session can then use |
preflight enumerates every one,
prints what it found, and exits non-zero when a hook definition or a non-empty
permissions.allow is present — or when a channel could not be read. It also
names every settings key it does not model, because an all-clear over the rest was
itself a finding in this kit's own audit history.preflight exists because a config loader is the one channel that rule cannot reach.
Two tools, one pipeline
One tells you what is wrong. The other tells you how long it has been wrong.
They share a single report format and stay in sync — but neither depends on the other to run.
The assessment engine
- Recon → nine specialist lenses → a non-negotiable verification gate
- Every finding labelled by evidence tier; retractions stay in the report
- Deterministic supportability metrics measured from git and your toolchain
- Branded
.docxwith a live TOC, index and page numbers — plus PDF, HTML and SARIF
The history dashboard
- Shows when each finding's line was first written, by whom, in which commit
- Timeline, accumulation plot, file treemap and lens icicle — all cross-filtered
- Pins to the exact commit the report names; refuses to guess a stale location
- Re-derives the verification ratio and flags a report that disagrees with itself
assessment.md → audit-dash → a single HTML file you can open anywhere
A defensible verdict, not raw scanner output.
Scanners and agents are excellent at locating candidates across a large codebase. Assay uses them for exactly that, then makes a human confirm each high-severity finding in the source before it ships.
Every finding carries its evidence tier, and retractions stay in the report — because a report with no retractions either got lucky or never verified.
The count is published too, so the discipline is legible to whoever reads the report rather than something you have to take on faith.
Capabilities
Everything needed to measure, verify and ship a verdict.
Adjudicated findings
Every finding source-verified before it ships and labelled with its evidence tier — retractions and all.
Deterministic metrics
Repo age, velocity, application LOC, bus factor, churn hotspots, cyclomatic complexity, coverage, mutation, vulnerable dependencies and an AI-authorship signal.
Language agnostic
Node, Python, .NET and Go out of the box; any language via lizard complexity. Anything it can't run, you pass in — LCOV, SARIF, audit JSON.
Word, PDF & HTML
A .docx with cover page, live clickable TOC, index and page numbers — no pandoc, no LaTeX. Same layout to PDF, or standalone print-ready HTML anywhere.
Corporate branding
Brand at the shell: pour the report into your own .docx template so its theme, cover, header and footer genuinely drive the document.
Scanner orchestration
secscan normalises Semgrep, gitleaks, Trivy and OWASP ZAP into one severity-sorted worklist — every row pending until a human adjudicates it.
SARIF & readiness
Findings as SARIF 2.1.0 for GitHub code scanning or Defender, evidence tiers preserved. A weighted readiness score rolls the verdict into one number.
Supply-chain screen
Two questions, never merged. A ⚖️ judged worklist asks is this the package you meant? of every direct dependency — a name one edit from a widely-used one is flagged, and the suggestion is confirmed on the registry before it is made. Beside it, a 🎯 measured check compares every lockfile hash against the digest the registry records. No tarball is ever downloaded.
Report conformance
verify-report checks a finished report against the template's own structure — required sections, table columns, the evidence vocabulary — and exits 1 under --strict. A deliverable that drifts from the format its claims depend on fails the build, not the client meeting.
Measured AI cost
What each lens pass consumed, counted from the run's own local transcripts — no network, and none of the audited code runs. With no session to identify, every figure reads "not collected" with the reason. It never prices the tokens: a rate table is not in the transcript, and a currency figure would be a judged number wearing a measured section's authority.
Continuous monitoring
A scheduled CI workflow re-runs the deterministic checks weekly and gates on only the objective signals — an end-of-life runtime, critical/high vulnerable deps, and a lockfile digest that contradicts what the registry published. Three exit codes, not two: pass, breach, and withheld — so a metric that went ungraded can never be read as one that passed.
Self-verifying build
The build checks every heading against the generated TOC and throws if one is missing. The kit holds itself to the standard its reports demand.
The second engagement
Did last year's fixes actually hold?
Most assessments start from zero every time, so a re-audit cannot tell you whether anything improved. Assay keeps a ledger — one record per finding — and the next report opens with a section it is not allowed to leave blank.
Reconcile
fixes turns every finding in the report into its own record, and flags any that
resembles a finding a previous engagement already raised — a reopen candidate, put
in front of a human for a decision.
Status
progress renders Since the last audit: what the previous engagement
raised, how much closed, what reopened, what this one adds. It anchors on the ledger and the
report's own scope line, and refuses to guess which two audits you meant.
Re-test
regress re-runs every probe that names a closed finding. Status is what
somebody wrote down; this is whether it still holds. Broken leads the section whatever
its count, and the command exits 1.
Unproven is not a failure. It is a fact about the evidence.
A closed finding that no probe names is resting on a sentence someone wrote on the day of the fix and never re-examined. That is not the same as broken, and reporting it as broken would be its own false finding — so it gets its own bucket and its own count.
Measured on this kit's own ledger: 27 of 103 closed claims had no probe naming them — 10 of them critical or high. The discipline is worth commissioning precisely because it finds that in its own record rather than yours.
regress is gated on explicit consent: the default is no, and in an
agent session with no terminal it refuses and does not run them. Where consent was not
given the section says so — a blank would read as "nothing to flag".The finding-history dashboard
The report says what is wrong. This says how long it has been wrong.
Every finding lands on the commit that first wrote its line — so you can see whether your risk arrived in one bad fortnight or accreted over three years. One self-contained HTML file that computes every chart in the browser. No server, no build, no CDN.
The commit is not optional
Line numbers mean something only at the commit the report names. The dashboard pins to it and stops if the checkout doesn't contain it — rather than blaming HEAD and inventing a plausible, wrong answer.
Stale locations reported, not guessed
A location git can't resolve gets git's own reason shown in the row and counted in a banner. That's a finding about the report, and more useful than a blank.
Filters scope every view
Filter by severity, evidence, lens or churn hotspot and the KPIs, both plots, the table, treemap, icicle and churn chart all re-render together — so the numbers always agree.
Derives, then compares
It recomputes the verification ratio from the findings table and raises a banner when the report's prose disagrees with its own numbers.
Colourblind-safe by design
Severity rides position, not colour, and every plotted value is repeated in the table. A report whose whole purpose is not to mislead cannot ship a chart that does.
Churn, overlaid
Commits touching each file up to the assessed commit — high churn × high complexity made visible as the next likely defect site.
No data egress
Bring your own model. Your source never leaves the building.
The kit's own code calls no LLM. The mechanical half — scan, secscan, readiness, attest, build, the dashboard — is pure deterministic Node. The intelligent half is done by an agent driving a model you choose, and it is deliberately not locked to one provider.
| Tier | You run it with | What leaves your network |
|---|---|---|
| Turnkey | The /audit skill | Prompts + code excerpts to your model provider |
| BYO key | A thin runner on an agent SDK | Prompts + code excerpts to your provider |
| BYO any model | Any agent runtime, or by hand | Nothing — on a self-hosted model |
Deliverables
Questions
What people ask before commissioning a codebase assessment.
If yours isn't here, ask it directly — the answer usually depends on the codebase.
How is this different from running a SAST scanner like Semgrep or Snyk?
A scanner locates candidates and ranks them by pattern confidence. Assay uses scanners and LLM agents for exactly that reach, then adds the step they cannot do: a human confirms every high-severity finding in the primary source before it enters the report. Each finding ships with an evidence tier saying how it was established, and retracted findings stay visible in the report.
Can Assay be used for technical due diligence on an acquisition?
Yes — it is one of the two things it is built for. A due-diligence assessment covers security, code quality and long-term supportability, and joins every finding to the commit that first introduced it, so an acquirer can see whether the risk arrived in one bad quarter or accumulated over years. The deliverable is a Word or PDF report a deal team can circulate.
Can the report be used as evidence in a customer security review or compliance audit?
That is what the verification gate is for. Every finding is labelled live-exploited, source-verified or agent-reported, and the report carries a verification attestation showing critical/high coverage and the number of retractions. An auditor can see which claims were confirmed against the source rather than taking a tool's word for it. Findings also export as SARIF 2.1.0 for GitHub code scanning or Microsoft Defender.
Does our source code leave our network?
It does not have to. The toolkit's own code calls no LLM and is locked to no provider. The deterministic half — measurement, scanner orchestration, readiness scoring, report building and the dashboard — runs entirely locally, and "audit-kit scan --offline" makes provably zero network calls and can be run air-gapped. If you point the agent half at a self-hosted model, nothing leaves the building at all.
Which languages and stacks are supported?
Node, Python, .NET and Go are supported out of the box, and any other language through lizard complexity analysis. Anything the toolkit cannot run itself you pass in as data — LCOV coverage, SARIF findings, or npm, pip and dotnet audit JSON — so an unusual stack does not block an assessment.
What do we actually receive at the end of an assessment?
A branded Word document with a cover page, live table of contents, index and page numbers; the same report as a bookmarked PDF and as self-contained print-ready HTML; findings as SARIF for your code-scanning pipeline; an interactive finding-history dashboard that is a single HTML file with no server, build step or CDN behind it; and a ledger carrying one record per finding, so the next engagement can report what became of each one. The PDF is exported from the same laid-out document rather than re-rendered from markdown, so the hyperlinked contents survive as PDF bookmarks and the index stays clickable.
Is it safe to point an AI agent at a repository we do not trust?
Not by default, and that is the part most tooling ignores. An assessment runs with the untrusted tree as the working directory of a session holding shell and file-write authority, and part of that tree is consumed by the agent harness rather than read by a model: CLAUDE.md and AGENTS.md reach the system prompt, .claude/settings.json supplies pre-authorised permissions and hook definitions, .mcp.json declares tool servers and the commands that start them. Assay ships a preflight command that enumerates all twelve of those channels before you open the tree — plus anything unmodelled it finds beside them — and fails closed when execution is already pre-authorised, or when a channel could not be read at all. An unreadable channel is refused outright; a pre-authorisation can be accepted deliberately, with a flag or at a terminal, so the decision is always somebody’s. Separately, every prompt in the method binds the auditor to treat everything inside the target as evidence, never instruction.
Can a re-audit tell us whether last year's fixes actually held?
Yes, and it separates two things most re-audits merge. Status is what somebody wrote down when they closed a finding; whether the fix still holds is a different question and a different command. Assay keeps a ledger with one record per finding, reports what the previous engagement raised and what became of it, and then re-runs every probe that names a closed finding. Results land in three buckets: held, broken, and unproven -- no probe names it, so the evidence is a sentence written on the day of the fix. Unproven is reported as a fact about the evidence, never scored as a pass.
Do you check our dependencies for typosquatted or taken-over packages?
With the supply-chain option, yes, and the two halves are kept apart because only one of them is a measurement. The judged half asks whether each direct dependency is the package anyone meant: a name one edit away from a widely-used package is flagged, and the suggested alternative is confirmed on the registry before the suggestion is made. The measured half compares every hash pinned in your lockfile against the digest the registry records for that exact name and version -- no tarball is downloaded or hashed. It is a slopsquat and takeover defence, not a safety score, and it needs network access, so the offline mode refuses it outright.
What is a codebase security and supportability assessment?
It answers three questions about a codebase you did not write: what is wrong with it, how long it has been wrong, and how hard it will be to keep running. Assay covers it through nine specialist lenses — authorisation and isolation, injection and uploads, the LLM surface, backend correctness, frontend, infrastructure and CI/CD, leaked secrets and confidential data, OWASP cryptography, design and integrity, and maintainability — alongside deterministic metrics measured straight from git.
How long does an assessment take?
A depth flag scales the scope: --light for a first read, --thorough for a normal engagement, --full for the deepest pass. The lenses run in parallel, so wall-clock time is driven mostly by the human verification gate, which is never skipped at any depth. Reconnaissance on an unfamiliar repository is typically well under an hour.
Contact
Tell us about the codebase.
Whether it's due diligence on an acquisition, an inherited system nobody owns any more, or a security review you need to be able to defend line by line — start here.
We reply to everything, usually within two business days.