How to Build a Jarvis for Your Mac with Shortcuts, AppleScript and Raycast

Ask ChatGPT what to use if you want to talk to your Mac and have it do things — open apps, run a command, find an email — and it will not name a product. I tried it this week in a chat with memory turned off, and got this back:

if your goal is "Jarvis for my Mac", I'd build around Raycast/Shortcuts + an AI model + tightly permissioned macOS automation, rather than giving an AI unrestricted Terminal access.

That is reasonable advice. It is also a build, not a purchase, and nobody tells you where the build stops working. So this is the honest version: the stack, with code that compiles, and then the four places it falls over.

You already have most of it#

Before installing anything, check what shipped with your Mac:

which osascript shortcuts
# /usr/bin/osascript
# /usr/bin/shortcuts

Both are there on every modern macOS. osascript runs AppleScript and JavaScript for Automation. shortcuts runs anything you have built in the Shortcuts app, from the terminal, which is the bridge most people miss.

shortcuts run "Daily Standup" --output-path /tmp/out.txt

That is the real syntax — run, list, view and sign are the whole CLI. It takes an input path, an output path and a UTI for the output type, and nothing else. Small surface, but it means any Shortcut you can build with the graphical editor becomes a shell command, and anything that can run a shell command can now run it.

First, the part nobody mentions: macOS already ships a voice controller#

Before you build anything, know that Apple splits this job into two features and most people only meet one of them.

Dictation is the one you know: speak, text appears, and that is all it does. Voice Control is a separate feature in Accessibility that operates the machine. Apple's own examples are "Open Mail," "Scroll Down," or "Click Done". It covers window management, app switching, a numbered grid for clicking anything on screen, and recorded multi-step commands of your own.

It also runs locally. From Apple's Voice Control technical brief: "Voice Control audio processing happens on-device, so it works online or offline and keeps personal information private" (PDF).

Every "can I control my Mac by voice" thread should start here and almost none do. Turn it on in System Settings → Accessibility → Voice Control, say "Show commands" to see what is available in the app you are in, and give it ten minutes before writing a line of AppleScript. If your list of wants is "open things, click things, move around", you may already be finished.

Two catches. It is command-driven rather than conversational — you say "Command Mode" and "Dictation Mode" to switch between the two — so you are learning its vocabulary rather than talking naturally. And it takes standard Dictation away while it is on. Apple is explicit about this: "When Voice Control is on, you use Voice Control to dictate text; standard macOS Dictation isn't available."

That is the ceiling of what is free and built in. Everything below is about raising it.

Layer one: AppleScript, for apps that speak it#

AppleScript is thirty years old and still the most direct way to make a Mac application do something. Three examples that compile — I checked each with osacompile before publishing, which validates syntax without executing anything:

tell application "Mail"
  set theMessages to messages of inbox whose read status is false
  return count of theMessages
end tell

tell application "System Events"
  set frontApp to name of first application process whose frontmost is true
end tell
return frontApp

on run argv
  set theQuery to item 1 of argv
  tell application "Safari"
    activate
    open location "https://www.google.com/search?q=" & theQuery
  end tell
end run

Save the last one and it takes an argument: osascript search.applescript "flights to seattle".

If AppleScript's syntax makes you unhappy — it reads like a memo from 1993 — the same automation is available in JavaScript:

osascript -l JavaScript -e 'Application("Safari").activate()'

The catch is coverage. AppleScript only reaches apps whose developers wrote a scripting dictionary. Apple's own apps are well covered. Slack, Notion and most of the Electron generation are not, and no amount of cleverness fixes that.

Layer two: Shortcuts, for the apps that don't#

Shortcuts is the modern replacement, and its advantage is that app developers opt in with far less work, so coverage is broader among newer apps. Build a Shortcut in the editor, then call it from anywhere:

shortcuts list                    # what you have
shortcuts run "Clear Inbox"       # run one

The pattern that makes this useful for an assistant: build small, single-purpose Shortcuts with clear names, then let the model choose which to run. You get a safe, enumerable action space instead of an open shell — which is exactly the "tightly permissioned" part of the advice above, and it is the right instinct.

Layer three: the model#

Now you need something to decide which script to run. Two honest options.

Raycast AI is the polished one. Its AI Extensions turn plain language into actions in specific apps — Raycast's own examples are things like "block my day from 4pm" or "move all pdfs on my desktop to the trash". Worth being precise about what this is: Raycast AI does not autonomously run shell commands or plan multi-step work. It picks an extension and invokes it. That is a deliberate design choice and a good one for safety, but if your mental model is "it will figure out the steps," it will not. raycast.com

Open Interpreter is the other end. It is now a Rust coding agent with 67.9k stars, Apache-2.0 licensed, that runs commands inside native sandboxing on macOS, and supports exec, MCP, skills, hooks and permissions. It will write and execute code to accomplish a goal. It is also a coding agent by design and by name, which means its centre of gravity is a terminal and a repository, not your calendar.

Between the two sits the thing most people actually want and neither quite is: a resident assistant that hears you, decides, acts across your apps, and is still there tomorrow.

The permission wall, and where it actually is#

Here is the part that surprises people, and it is worth understanding before you spend a weekend on this.

A script that only talks to itself runs with nothing in its way:

osascript -e 'return "hello"'      # runs, no prompt, no permission

The wall appears the moment your script addresses another application. That is an Apple event, and macOS gates it. The app sending the event must declare NSAppleEventsUsageDescription in its Info.plist, and the user gets a one-time consent dialog per target app.

You can see this on your own machine. Apple's Keynote declares:

This will let Keynote control other applications. You can change this later in System Preferences.

Every app that automates another one carries a string like that — Pages, Numbers, iTerm, Sublime Text, Obsidian and Zoom all ship one. Grants live in System Settings → Privacy & Security → Automation, and they are per pair: this app controlling that app.

The practical consequence for a DIY build: when you run a script from Terminal, Terminal is the app asking for permission. Run the same script from a different launcher and the prompt starts over, because it is a different pair. This is the single most common reason a home-made setup works on Tuesday and fails on Thursday.

There are three more gates worth knowing, each separate and each in the same Privacy & Security pane. Accessibility is what lets something click and type into other apps, which is how you reach software with no scripting dictionary at all. Screen Recording is required to read what is on screen. Full Disk Access covers Mail's data store and other protected locations. They are independent — granting one grants nothing else.

Where the DIY stack stops#

I want to be fair to the approach: for a fixed set of tasks you do every day, scripts plus Shortcuts plus a hotkey is excellent, and it will outlast every AI product mentioned here. If that is your goal, stop reading and go build it.

It stops being enough at four specific points.

One: permissions are invisible until they fail. There is no manifest, no health check, nothing that tells you the Automation grant for Mail lapsed after an OS update. You find out when a script silently returns nothing, usually while you are relying on it.

Two: nothing remembers anything. Each osascript invocation is a cold start. Your scripts do not know who Sarah is, which project you meant, or what you asked for yesterday. You can bolt on a file of context, and then you have started building the boring 80% of an assistant — retrieval, ranking, staleness — instead of the fun part. We wrote about the shape of that problem in what a real Mac agent looks like.

Three: nothing resumes. A shell script dies when the terminal closes and does not come back when the Mac wakes. Anything you want to happen at 7am while you are asleep needs a launchd agent, error handling, and a plan for what happens when the machine was shut. That is a daemon, and now you maintain a daemon.

Four: you cannot tell what actually happened. This is the one that bites hardest. An AI layer reports success because the script exited zero, not because the email left. We wrote a whole piece on this failure mode — your agent will tell you it sent the email — and it is the reason a scripted setup is fine for reversible work and genuinely risky for anything that sends, deletes or pays.

So which should you do?#

Our bias here is obvious and worth stating plainly: we build one of these. What we would not tell you is that the DIY route is a waste of time, because it is not. Most people who go and script the five things they actually do every day never need anything more, and the ones who do arrive at a product knowing exactly what they want from it.

If you want the scripted route done well, our Mac voice commands cheatsheet has the ones that survive contact with real use. If you want to skip to the resident version, Jarvis is free and open source, runs on-device by default, and speaks MCP so it reaches the tools Shortcuts never will.