Multiple Choice Is Not a Decision: Teaching a Planning Agent to Ask More Questions

I opened PLAN.md on a new project last month. That is the planning doc every project in my setup carries, and its decisions section is supposed to be the record of what the project has decided. Mine was still the blank template: two bullet points under the heading, and one of them was about where to put the output folder.

This was not a project where nothing had been decided. I had been going back and forth on it for a week. The decisions existed. They were in a chat log somewhere, or in my head, or in the shape of the code I had already written. They were not in the document whose entire job was to hold them.

I went and looked at the skill that owns these docs and found the real problem. It has four workflows, and every one of them reshapes material that already exists: create the files, retrofit them into a repo that has none, compact a doc that got fat, clean finished work out of the task file.

Not one of them generated any of the files.

• Multiple Choice Is Not a Decision
• Then Matt Pocock Wrote It Down in 28 Lines
• It Ends in a Conversation. My Docs Have to Outlive the Conversation.
  • 1. The Round Ends in a Disposition, Not an Answer
  • 2. It Is Re-Entrant
  • 3. The Files Append, They Never Get Rewritten
  • 4. The Relentlessness Had to Go
• It Is a Utility, Not a Stage
• What I Would Tell You

Multiple Choice Is Not a Decision

Here is the part I had been doing wrong on my own, well before any of this got written into a skill.

When you plan something with an agent, the conversation drifts into a shape almost immediately. You describe what you want. It comes back with options. A, B, or C, with a short paragraph on each, and sometimes a helpful little table. You pick the one that is least wrong, it says great choice, and you move on to the next menu.

Nobody argued with you, nothing got stress tested, and the option you picked was picked because it was the best of three things a model generated in two seconds.

I got tired of it. What I actually wanted out of planning was an argument, where I have to say why and get pushed on the answer.

So I had been doing a crude version of this by hand. Telling the agent to stop offering me options and start asking me questions. It sort of worked, the way anything sort of works when you are improvising it fresh every session with no structure behind it.

Then Matt Pocock Wrote It Down in 28 Lines

The idea I built on is not mine. It is Matt Pocock’s grilling skill and I want that up front rather than buried in a credits line at the bottom, because the core of it is his and it is the good part.

It is 28 lines. 319 words. It opens with this:

Interview the user relentlessly until you reach a shared understanding.

Then it gives you the structure that makes that possible, which is what I had been missing. Model the conversation as a design tree, where every decision branches into the decisions hanging off it. Work the tree in rounds. The frontier is every decision whose prerequisites are already settled, which is to say the questions you can ask right now without guessing at answers you have not heard yet.

Two rules in there are doing most of the work.

The first is ordering. A question whose answer depends on another question still open in this round belongs to a later round. That is the whole technique, honestly. Violate it and you get answers the user has to retract two rounds later, which is exactly what my hand-rolled version kept doing.

The second is that every question ships with the agent’s recommended answer, so the cheap reply is “yes to all but Q3.”

And finding facts is the agent’s job. If a question needs to know what is in the filesystem or what some dependency actually does, it goes and looks. The decisions stay yours.

The session is done when the frontier is empty: every branch visited, nothing left assumed.

I read it and wanted to use it, and immediately hit the thing that made it not fit.

It Ends in a Conversation. My Docs Have to Outlive the Conversation.

Grilling ends at a shared understanding held between two parties in a chat window. That is a perfectly good place for it to end if you are making one decision today.

I am not. These projects run for months. The entire premise of this whole setup is that the documents are the memory, because the agent’s context window is not. An interview that produces nothing but a shared understanding produces nothing at all when the session ends.

So four things changed on the way in.

1. The Round Ends in a Disposition, Not an Answer

The frontier does not empty because everything got answered. It empties because everything got filed. Every node in the tree closes as exactly one of four things:

The line in the workflow that explains why is the one I would keep if I had to throw the rest away:

A blind spot is an unasked question, not an unanswered one.

“I do not know yet” is an acceptable outcome. It becomes a file. What is not acceptable is a branch nobody ever walked down.

It writes at every round boundary for the obvious reason that I stop things halfway constantly and a workflow that only saves on completion would lose everything every time I do.

2. It Is Re-Entrant

The upstream is a one-shot interview. Mine has two modes.

Greenfield is the one-shot case: there is no planning doc yet, so the tree starts at the root; with nowhere to file, nothing gets written during the session, and at the end it hands the whole disposition to the scaffold workflow, which emits a PLAN.md that is already populated. If you quit early it scaffolds with whatever you settled anyway. Three decisions and six open questions beats an empty template.

Continuing is the one I actually use. It reads the existing planning doc and the docs/ tree and seeds the design tree from them before asking anything:

• existing decisions become settled nodes, pruned rather than re-litigated, and each one pushes the frontier outward because the questions hanging off it are exactly what just became askable
• open questions come back as frontier nodes carrying whatever partial answers previous rounds accumulated
• parked items come back only if something settled since they were parked makes them answerable now, because dragging every parked idea back every session is how a parking lot turns into noise you learn to skip

This turns the docs into a breadcrumb trail instead of a stack of unrelated one-shot interviews.

3. The Files Append, They Never Get Rewritten

A Qxx.md touched by a later session gets a new dated section added under the existing ones.

I had to put that in as an explicit rule because the instinct to tidy is strong. That growing, slightly repetitive, occasionally contradictory file is the record of what you thought in June that made the July answer obvious. Rewrite it into a clean summary and you’ve thrown away the reasoning.

4. The Relentlessness Had to Go

This is the one I keep coming back to.

“Relentlessly” is right there in the first line upstream, and for a one-shot session it is correct. A grill that gives up when you get tired is not doing its job.

But relentless plus re-entrant is just nagging. If the thing can come back next week, it does not need to squeeze everything out of you today. So the stop rule is written as an invariant rather than a preference:

Stopping is never negotiated.

No “are you sure”, no “we have not covered X yet”, no one more round, and specifically no listing what I am about to miss, because listing what I am about to miss is just arguing with extra steps. If I name a branch, it parks that branch and keeps going elsewhere.

The reason that is safe is the rule from change one. Everything unvisited gets filed on the way out.

Nothing is lost by stopping, which is precisely why stopping needs no defense.

It Is a Utility, Not a Stage

Every other route in this skill is triggered by what the repo looks like. No planning doc means grill first, then scaffold. Planning doc but no task file means adopt. Fat planning doc means rebalance. Task doc full of finished work means evict.

The grill has a row in that table that is not a filesystem state at all:

It fires on what I am doing, not on what the directory contains. Two entries: I ask for it, or I am clearly thinking out loud with no question attached and it offers in one line and waits. That second one has a rule attached. One line, then shut up.

That is why it sits at different points in the process rather than at the front of it. It is not step one of planning. It is the thing you reach for at any point where you are about to settle on an answer.

What I Would Tell You

If you want the idea, go read Pocock’s 28 lines rather than my version. It is the good part, it is short, and you will be running it in five minutes.

The next post is the last one in this series, and it will be: the whole thing, start to finish, how it actually works, so nobody has to read a skill directory to figure it out.