Lacunari has a reasonable test suite. It also has a history of bugs that reached real use, and when we lined them up, every one belonged to one family. The family is worth naming, because a test suite is structurally bad at catching it.
The exhibits
psql exits 0 on an aborted transaction
Without ON_ERROR_STOP=1, psql runs every statement in a file, reports nothing useful, and exits zero — even when the transaction aborted at statement three and everything after it was discarded.
So a migration whose transaction aborted is indistinguishable from a clean install. That shipped. The entire suite then ran happily against a schema silently missing a whole layer, and passed, because every test that would have touched the missing layer was in the part that never got created.
CREATE OR REPLACE VIEW can only append columns
Add a column in the middle of an existing view and Postgres refuses, aborting the migration — which, per the exhibit above, looked like success. Same incident, second cause.
flock does not exist on macOS
The drainer used flock for its lock. On BSD it is simply absent, the invocation failed, and the failure aborted the run silently. The drainer never processed a single task on a Mac and never said so. Now it is an atomic mkdir with stale-PID recovery.
An operator with no power
The keeper was granted membership in a BYPASSRLS role so it could release other agents' stale claims. But Postgres never inherits role attributes through membership — only privileges. The keeper connected fine, ran its queries fine, and released nothing, ever. It reported success every time. Policies now test with pg_has_role().
An audit log that named the wrong actor
The trigger was SECURITY DEFINER, and inside such a function current_user is the function's owner rather than the caller. Every action in the immutable audit trail was attributed to postgres. The log was working perfectly and telling us nothing.
A lock that guarded the honest path
Claims locked the files a task declared. Changesets wrote to those same files and never consulted the locks. The guarantee could be walked around by simply not claiming anything first — the lock stopped the correct workflow and left the other one open, which is the worst kind of boundary: one that makes people believe they are protected.
Why tests miss this
A test asserts that an operation reports success and that its visible effects are correct. Both halves of that sentence are the problem.
These bugs report success — that is their defining property, so the first half of every assertion passes. And their effects are usually invisible at the scope the test runs at: the missing schema layer is only detectable if you look for something in it, the keeper's impotence only shows up after a lease expires on a machine that has been running for hours, the audit attribution is only wrong when more than one identity exists.
Worse, several of these bugs hid each other. The aborted migration meant tests never exercised the layer where the next bug lived.
What actually catches them
- Assert on artifacts, never exit codes. Do not check that the migration returned zero; check that the table exists. Do not check that the extractor ran; check that rows appeared.
- Test the refusals. A security control that stops working almost always keeps exiting 0. Every rule that is supposed to deny something needs a test that it still denies it.
- Run it unattended for a week. Most of these only surface with real time and real crashes. A soak on a cheap machine found things eleven hundred assertions could not.
- Make the environment self-verifying. Our soak box is pinned to a commit with a manifest, and every daily health line reports whether the checkout still matches. Otherwise "it ran for a week" is a claim about a build you cannot identify.
The honest version of a passing suite
We publish the assertion count, but it is not the number worth reading. The number worth reading is that every bug real usage found was invisible to all of them — and that the response was to change what we assert on, not to add more assertions of the same kind.
It is also why this project's releases are gated on unattended runtime rather than a green check. A green check is exactly the signal this entire class of bug is best at producing.