I left a crew of Hermes Agent workers running unattended overnight to get through a multi-step research job. By morning I had a list of nine ways it had broken. None of the failures were exotic. Most were the kind you only find by leaving the thing alone for eight hours.
The setup
One orchestrator agent routes work to about a dozen role agents (researcher, analyst, docs writer and others) through a kanban board. A dispatcher inside the Hermes gateway starts one worker per card.
Reasoning and web research run on Claude through a subscription. Mechanical work (filing, formatting, arithmetic) runs on a local qwen 27B model on one RTX 4090 through Ollama. When Claude hits its usage limit, agents fall back to the local model. A small control script with no model calls of its own starts, pauses, resumes and reports on a run.
A full overnight run was about 70 cards. The size that actually held up was closer to 27.
What went wrong
Roughly in the order I hit it.
The subscription limit went in about 12 minutes
Three frontier-model workers plus the orchestrator all hit the Claude session limit (HTTP 429) almost immediately. Every card defaulted to the strongest model available, so four agents were drawing on the same quota at once.
Fix: choose the model per card, not per crew. Mechanical work goes to the local model by default, and the top-tier model is off unless a card asks for it.
Fallback ran out of context
One card fell back to the local model, which has a 64K context window, and died at 65,464 of 65,536 tokens. Its brief told it to read an entire repo. A frontier model with a large window shrugs that off; a 64K local model cannot.
Fix: keep cards small. Each card body is a parameter block plus a pointer to a template, every output file carries a word cap, and each card does one thing.
A card waited forever for a comment
An agent with no file access was told its brief would arrive as a comment from the orchestrator. The comment never came, because it depended on a human-driven step happening first. The card just sat there.
Fix: the card that creates downstream work writes the full brief into the new card’s body, not into a comment. Workers can create cards for other agents directly.
The concurrency cap did nothing
I set a cap of 4 concurrent cards and watched 9 start at once. The gateway’s embedded dispatcher reads that setting once at startup, and it reads it from the config of the profile the gateway itself runs as, not the orchestrator’s. I had set the right number in the wrong place.
Fix: set it where the gateway actually reads it, and restart the gateway between runs so the new value takes.
Three cards stalled on approvals nobody was there to give
A template told an agent to do arithmetic through code execution. Each call sat waiting about six and a half minutes for a human approval that wasn’t coming at 2am, then timed out. The agent correctly blocked instead of guessing. The run stopped around 3am, waiting on approvals I was asleep for.
Fix: templates never depend on a tool that needs a human in the loop. Arithmetic is written out in the output file with every formula shown, so there is nothing to approve.
Fallback saved the run but hurt the quality
In a longer run, 8 of 10 reasoning cards fell back to the local model, and the output was noticeably weaker than what the frontier model produces. Fallback keeps a run moving. It is not a free substitute for the model you wanted.
Fix: smaller runs, plus a morning digest listing which cards fell back so they can be queued for a re-run on the stronger model.
An agent refused its card as outside its role
One agent looked at a legitimate card and judged it a misroute, because its role definition was narrow enough that the card didn’t obviously fit. The routing was correct. The role description just didn’t say so.
Fix: state the card’s scope on the template and note that the routing is deliberate, so the agent doesn’t have to guess whether it is being asked to step out of bounds.
A hand-made re-run wrote into the wrong run’s folder
Card titles repeat across runs. I built a re-run by matching on title, which picked up paths from an older run that shared one. Separately, a card wrote a stray copy into the repo root through a relative path.
Fix: filter by run id, never title, and make every output write use an absolute path so a card can’t land somewhere by accident.
Pause could lose work
Moving a running card to the scheduled state drops its worker’s claim on that card, which can throw away work in progress.
Fix: only pause cards that are still queued, let running cards finish, and re-sweep each minute to catch anything they release once done.
The pattern
| Failure | Fix |
|---|---|
| Quota exhausted in 12 minutes | Model chosen per card |
| 64K context overrun | Small cards, word caps |
| Brief waiting on a comment | Brief written into the card body |
| Concurrency cap ignored | Set in the gateway’s profile, restart |
| Stalled on approvals | No human-in-the-loop tools in templates |
| Weak output after fallback | Smaller runs, morning fallback digest |
| Role refused a valid card | Scope stated on the template |
| Re-run wrote to wrong folder | Filter by run id, absolute paths |
| Pause dropped work | Pause queued cards only |
None of these needed a rewrite. They needed the attention you only pay once something has failed in front of you. The rules I would apply to any crew left running unattended:
- Assume the model will be throttled or swapped out, and design for it before you need it.
- Keep cards small: one item, one output, one word cap, and no instruction that assumes a huge context window or a human watching.
- Check where a setting is actually read from, not where you think you set it, and restart whatever needs to pick it up.
- Treat pause, resume and re-run as real operations with their own failure modes, not afterthoughts bolted onto a dispatcher built only to move forward.
The run that held up was 27 cards against 70. That is probably the most useful number in this whole list.