Reminding yourself to check does not make you check

The rule held exactly as long as I was already checking

The setting is my public screening runs: an agent reads public filings, listings and registry pages, returns candidates, and cites what it read. Nothing confidential enters; the task is to sort a wide field of public signals into a short list I then work by hand. The rule was simple: before a candidate moves forward, open the cited sources.

It held while I was still finding things to correct. Then the outputs started coming back clean, one after another, and somewhere in that streak I moved a candidate forward without opening anything. I have no count of the outputs or days the rule survived, and I cannot name the output on which it first failed – the log of the rule depended on the attention the rule was supposed to discipline.

I first read this as a personal lapse. The oversight literature reads it differently: reviewing and correcting an agent’s outputs after execution is the named phase of operator scrutiny, and it is a phase, not a permanent state.² Operators adjust trust moment by moment through interaction with automation, and a failure cuts trust hardest when the outcome is one nobody wanted.³ Read backwards, that is the mechanism: a run of good outcomes lifts trust one notch per output, and no rule sits between the notches.

The intuitive fix is to make the rule more explicit – write it down, remind yourself what calibration means. The direct test of that idea came back empty.

No improvement · in the accuracy of people’s reliance on automated advice after trust-calibration feedback, 2025 study¹

The participants who were told to calibrate their trust relied on the advice no more accurately than those who were not.¹ That is my rule in miniature. The rule was a piece of trust-calibration feedback addressed to myself, and it did what the feedback did in the study.

Only a check that fires on its own outlasts the streak

A systematic review of clinical decision support found that erroneous advice was 26% more likely to be followed than comparable advice under control conditions⁴ – a different domain from mine, but the same shape: when the system is wrong, the human tends to go with it unless something interrupts. In a screening run, the wrong candidate moving forward costs a morning; the cost climbs only when someone acts on it.

Two facts from the governance side explain why the rule could not have held:

1. In the operator governance model, the operator holds continuous authority over autonomy boundaries and makes the trust-calibration decisions on the basis of observed behaviour.⁵ Nothing else enforces those decisions. A rule I set is a rule only I can keep, and my attention is the single point of failure.
2. High-consequence workflows are described as needing event-triggered review in addition to scheduled review – scrutiny that rises on a trigger rather than on a fixed cadence or on memory.⁶ The category exists; my rule was neither scheduled nor triggered, it was remembered.

Put those together and the diagnosis is structural. An attention-based rule and the trust curve it is meant to counter run on the same input – consecutive unchallenged outputs – so the rule weakens precisely when it is needed most. A check that survives a long streak has to fire without the operator choosing to fire it, and it has to fire before the point where an output stops being a candidate and starts being a commitment.

What that check looks like in my own runs I am not going to specify here; the mechanics stay in the house. I also do not have a record of what followed the first skip – whether I reinstated the rule, tightened it or let it go – which tells its own story about how much the rule was ever doing.

The next check I add will not depend on my remembering to add it.