Articles / Method
Start from what you can’t afford to have happen
Why a checklist misses the way AI tools actually hurt a firm, and what a safety engineer checks instead.
Most reviews start with a list of controls and tick them off. That produces a gap list, and it misses the way these systems actually hurt a firm: the tool performing exactly as designed, and a person downstream relying on the output without checking it.
A model that invents a citation isn’t malfunctioning. It’s doing what it does. The accident happens in the handoff.
Start with the losses
So I start with the losses: privilege waived, a sanction, a malpractice exposure, a client’s data somewhere it shouldn’t be, a conflict breached. Then I identify the states of your system that would lead to one.
Some of those states are already the subject of written orders. A federal court in Colorado ordered that any party using AI with confidential information must “retain written documentation of these contractual protections.”1
Four questions for every control
For each of those states, I ask four questions:
- What control is meant to prevent this?
- Does it exist, in contract and configuration rather than intention?
- Would it hold at 11 p.m., with a rushed associate?
- How would you know it failed?
You don’t need to stop using these tools. You need to be able to show your work.