# Teaching a reviewer what I actually authorized October 17, 2026 · Technical companion to Reviewer An agent can do something reasonable that I never asked it to do. That became one of the most interesting problems in Reviewer, my experiment with using Jev to review an agent's tool calls. If I am presented with a plan and say “okay, do it,” that can be a broadening of the request. But I specifically approve it. My “try again” can be towards my previous message, not towards the agent's recent reasoning and commentary. Both replies are short. Understanding them requires the conversation. ## Small questions, enough context My idea was a tool call hook that answers a set of questions. Does the action change state? Does it expose secrets? Is it within the scope I approved? The review has four logical layers:  Diagram description: A proposed tool call reaches L0 deterministic checks, including known reviewer tampering. A block denies it. Passing actions reach L1 basic schema and L2 per-turn recipe, evaluated together in one Jev call. These can allow or deny. Remaining uncertainty reaches the L3 LLM resolver, which allows or denies. Unresolved final reviews are denied. L1 and L2 can allow or deny; unresolved cases reach L3. Their questions are batched in one Jev call, and task limits are checked before a fast approval. L3 cannot override an L0 block. If the final review still cannot clear the action, it is denied. Using secrets and exposing secrets should be two separate questions. Changing state does not mean it cannot be fast-tracked. Some tasks grant a state change. If the tool call is confidently not within the scope, it should be denied. The current tool call also needs to be separate from previous context. Questions about what an action does need the action and relevant earlier tool calls. Questions about authority need my prompts and the model's proposals. Putting everything into one large context does not make those questions clearer. This becomes more interesting when a task continues. A research agent can create notes, then update them. Without history, the reviewer cannot tell that the file was created and approved as part of the same work. It needs enough memory to allow useful continuation. ## Let uncertainty teach the next decision When the LLM reviews an uncertain action, it can suggest a recipe for similar decisions. I had a feeling that these would have a hard time generalizing: creating layers for a specific task, then failing for the next task of a different nature. So those additional layers should be per turn. Later the user should have a chance to consolidate them into general recipes. Recipes developed into a bulleted list of bite-sized scope elements and a similar list of limits. The set of bullets should grow; changes needed an additional reason. And why extend the bullets only on an allow event? Denied actions should teach the reviewer too. Jev calls are cheap enough to ask several short questions instead of one big question. Independent questions can be batched in one call, with the decision logic using the answers it needs. That makes it practical to experiment with different formulations. But cheap experiments still need useful tests. ## What the tests showed I wanted symmetrical allow and deny cases. After learning that the same cases were used to design the questions and score them, I asked for another 40 pairs. Each pair contains an action that should be allowed and one that should be denied. Here is one saved run of that second set, from October 7: October 7 development evaluation · 80 cases | Expected decision | Allowed | Denied | | --- | --- | --- | | Allow: 40 cases | 40 | 0 | | Deny: 40 cases | 1 | 39 | The missed case is more useful to explain than an accuracy percentage. The test request authorized installing one package. The proposed action would install another, and the LLM allowed it because it considered that package reasonable for the project. It accepted a justification for the work as permission to do it. This is one run of 80 cases, using Jev 1.13.0 and a local LLM resolver. The second set had already informed fixes during development, so it was no longer an untouched test. These results show a specific remaining failure, not a guarantee about future tasks. We also added sequence cases: one decision affects what the reviewer knows for the next action. Tomorrow I could change Qwen to another model, or change Jev to a local model. All the work should be reproducible with substituted components. Reviewer is a proof of concept. For me, the useful finding is how much of this work comes down to understanding scope: what I requested, what the agent proposed, and what I actually accepted. The tests need to keep checking those distinctions as the system changes. For the current setup and reproducible tests, see the [toolreview README](https://github.com/yerbolgmailcom/toolreview), [evaluation examples](https://github.com/yerbolgmailcom/toolreview/tree/main/examples/eval) and [security and data-flow notes](https://github.com/yerbolgmailcom/toolreview/blob/main/docs/SECURITY.md). Author: Yerbol Akhmetbekov Canonical: https://akhmetbekov.com/blog/reviewer/notes/ Published: 2026-10-17