Writing Tools for an Agent That Runs on Someone Else's Computer

Most writing about tool design for agents assumes the tool runs on infrastructure you control. You know the runtime, the network, the credentials, and the machine is awake.

We build an agent that runs on the user's Mac. Nothing on that list holds. The machine sleeps mid-task. The credentials are the user's own API keys, and may be absent. The operating system can refuse a capability at any moment and only tells you when you try. Native dependencies have to exist for two architectures. And the same tool has to work under whichever model backend the user picked this week.

Here is what that changed about how we write tools.

Tool definitions are rent#

Every tool you expose is serialized into context on every single turn. Name, description, full parameter schema. A registry of eighty tools is a standing tax on the model's attention and on the user's token budget, paid whether or not any of them are relevant.

The naive fix — fewer tools — is the wrong trade, because capability is the product. The fix that works is progressive disclosure:

• A small always-on set: the handful used in most turns.
• A search tool over the rest, so the agent can find run_shortcut when it needs it without carrying its schema through every conversation about the weather.
• A capability manifest tool covering things a tool search cannot express.

That last one deserves its own section, because it fixes a failure mode we did not see coming.

"Can you...?" is a routing problem#

Users ask agents what they can do. Constantly. And an agent asked "can you see my screen?" will answer from its own weights — which is to say, from a general impression of what AI assistants do, not from your registry.

It will say yes to things you have not built, and no to things you have. Both are expensive: the first produces a promise the runtime then breaks, and the second is the product refusing to sell itself.

So the capability question gets a tool of its own — one that returns Jarvis's actual manifest, including modalities that are not tools at all. Seeing the screen is an input channel; no tool search can surface it, because there is no tool named "eyes." The instruction attached is blunt: call this before answering a "can you" question and before refusing anything.

Write to the strictest provider, not the most permissive#

This is the portability trap, and it cost us a production incident.

Model providers disagree about what a tool schema may contain. Most ignore keys they do not understand. One does not: a realtime API we support validates tool declarations against a strict subset of JSON Schema and, on encountering a key outside it, closes the websocket rather than ignoring the field. The symptom was not a validation error. It was a voice session that reconnected in a loop, for reasons that appeared nowhere near the schema that caused it.

Two rules came out of that:

1. Author every schema against the narrowest dialect you support — a flat allowlist of fields, no oneOf, no $ref, no clever composition. If a provider needs richer structure, that is a per-provider adapter's problem, not the tool author's.
2. Sanitize at the boundary, by allowlist. Third-party tools arriving over MCP were written against nobody's constraints in particular. Strip to known-good keys and cap the declaration count before anything reaches a provider that will hang up on you.

The general lesson: a tool definition is a wire format with several incompatible readers. Treat it like one.

One tool, every brain#

Users can switch the model backend. When they do, every capability has to survive the switch.

We learned this the way everyone learns it. A scheduling feature was implemented inside one backend's tool loop, worked beautifully, and vanished the moment a user switched providers — not with an error, but with a confident "Done" from a model that had no such tool. The absence of the tool did not stop the agent from narrating its use.

The rule now: a capability is not shipped until it is reachable from every backend. In practice that means tools live in one registry behind one interface, and backends that cannot host them natively get them over a bridge. Any list of "tools this backend supports" that is maintained by hand will drift, and the drift is invisible until a user reports that a feature disappeared.

Composable primitives beat one tool per action#

The instinct when a user asks for something new is to add a tool for it. Do that for a year and you have a hundred near-duplicate tools, an unaffordable context bill, and a model that cannot tell send_email from email_send_draft.

The better shape is a small set of composable primitives the agent chains — read a document, call a connector, run a script — instead of a bespoke tool per action. Fewer, more general tools compose into things you did not anticipate, which is the entire promise of an agent. It also means the interesting surface becomes the skills written on top of those primitives, which can be authored, versioned, and shared without shipping a build.

Agent-authored code needs two independent layers#

Once an agent can write scripts or skills that persist and run later, prompt injection stops being a conversational nuisance and becomes a persistence bug. A poisoned web page or email can try to talk the agent into writing something malicious that then runs on a schedule, long after the conversation that produced it.

Our answer is two layers that do not depend on each other:

• Author time. A static scan before the text is ever written to disk, rejecting the obvious patterns — "ignore previous instructions", credential exfiltration shapes, destructive shell. It is a best-effort filter and we say so in the code: it cannot prove a script is safe.
• Execution time. A hard deny list in the shell layer, and a scrubbed environment — authored scripts run without the app's own API keys, so a script that escapes the first layer still cannot steal what it was most likely written to steal.

Neither is sufficient. A static scan is bypassable by anyone who reads it. A runtime deny list cannot stop a script that only does permitted things in an impermissible order. Two cheap independent layers beat one expensive layer that is claimed to be complete.

Smaller things that turned out to matter#

Thread cancellation everywhere. Every tool that can take more than a moment accepts an abort signal. A stop button that stops the narration but not the work is a lie with a network bill attached.

Assume the machine sleeps. A laptop closing is a normal event, not an error. Any tool that starts something durable needs its work to survive that, or the agent stops halfway through and never mentions it again.

Two architectures, always. Ship an arm64-only binary inside an Intel build and it dies at launch with an error the user cannot act on. We now check the Mach-O header rather than trusting the build to have done the right thing.

Descriptions are prompts. The description field is not documentation — it is the only instruction the model reads about when to reach for this tool. Write it as guidance to a new colleague: when to use it, when not to, what it will not do.

If you want to read the implementations rather than the summaries, Jarvis is free and open source.