// What was hard
Five problems a demo never hits.
Most of the fixes moved a rule out of the prompt and into code.
Ten out of thirty is a diagnosis, not a score. The June live sweep started at 10 of 30 prompts, failing for a mix of reasons. So each failure got a cause before it got a fix: tool routing, a model breaking the answer schema, prompt wording, a too-literal test, or the provider being down. Fixing the layer that was actually wrong took the suite to 30/30 the same day. Behind it now: 502 automated tests, plus the 31-prompt live suite, which is run by hand.
The model talked its way around a guardrail. Asked to email an outside address, it refused in words, then reread the request as "send it to me" and called the email-preview tool anyway. A sterner prompt wasn't the fix. Now a literal address in the conversation removes both email tools from the model's list. Rules are enforced by which tools exist, not by what the model promises.
Good at tools, bad at forms. DeepSeek V4 Pro chose tools well but kept breaking the structured-answer schema. Rather than drop it, the catalog grew capability flags and DeepSeek moved to plain-markdown answers; seven failing prompts passed on rerun. A schema-breaking answer now marks its model degraded.
Models don't know when they're done. In forced final turns, some printed their own tool-call markup or narrated a plan instead of answering; others quit after the setup tools. One diagnostic run signed off with "The answer is 42." Now wait rules hold the loop open until the evidence exists — a root cause needs a correlation or diagnostics tool, an email a preview — and a stream filter drops anything before "## Answer."
A timeout must never cut an email in half. Read tools stop at 120 seconds; the five write tools are exempt, because a half-sent email is worse than a slow one. Sends are claimed atomically, recipients are rechecked on the server, and each outcome is recorded as Sent, Failed, or Uncertain — the last because sometimes that's the honest answer.