> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rayline.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Local mode

> Route turns between your own inference servers and Rayline Platform, and optionally make routing decisions on your machine.

Use Rayline CLI’s local mode when you want an inference model on your hardware to handle selected work. This is separate from Rayline CLI’s cloud mode, the default, which configures tools without running a local process. Start with the [local mode guide](https://github.com/rayline-ai/rayline-cli/blob/v0.5.4/docs/local-mode.md); model memory and runtime requirements depend on your configuration.

> [!NOTE]
>
> Local mode and the local router are separate choices. **Local mode** (`--local`) runs a Rayline process on your machine that sends each turn to your own inference server or to Rayline Platform. The **local router** additionally makes ARC's routing decision on your machine; it needs an Apple Silicon Mac, or an NVIDIA GPU on Linux or Windows x86\_64. Without one, use local mode with the hosted router, the default.

```bash theme={"system"}
rayline run claude --local
rayline local status
```

## Choosing a local model

Rayline CLI doesn’t run or download the inference models that answer your requests. Run the model on a server you manage, then list the server and the model in `~/.config/rayline/settings.json`, with the routing policy:

```json theme={"system"}
{
  "endpoints": {
    "gpu-box": {
      "base_url": "https://gpu-box.example.com:8443",
      "protocols": ["anthropic_messages"]
    }
  },
  "models": {
    "z-ai/glm-5.3-flash": {
      "endpoint": "gpu-box",
      "model": "GLM-5.3-Flash-EXL3",
      "context_window": 1000000,
      "max_output_tokens": 32000
    }
  },
  "routing": {
    "decisions_model": "arc-swift-1.0",
    "classifier_model": "anthropic/claude-haiku-5.5",
    "policy": { "preset": "arc-policy.prefer-local.v1" }
  }
}
```

Check the server’s protocol and model capabilities against the [local mode guide](https://github.com/rayline-ai/rayline-cli/blob/v0.5.4/docs/local-mode.md), which describes the full format. See [Rayline for agents](https://docs.rayline.ai/tools/custom-agents.md) for SDK examples.

### All local commands

| Command | Purpose |
| - | - |
| `rayline local start [--config <file>] [--router <local\|hosted>]` | Start the local process. `--router` sets where routing decisions run. |
| `rayline local status [--json]` | Show whether the local process is running. `--json` prints it for other programs. |
| `rayline local stop` | Stop the local process. |
| `rayline top [--json]` | Show live request metrics from the local process, and where routing decisions are made. |

## Hybrid by default

`rayline run claude --local` sends every model request through the Rayline process on your device. ARC picks the model for each request according to your [routing policy](https://docs.rayline.ai/arc/routes/hybrid.md). The process sends the request to your inference server or to Rayline Platform.

Your cloud provider may still need credentials and charge for inference.

Only model requests go through the local process. Claude Code's other network traffic, such as MCP servers and web requests, still goes to its usual destinations.

Local mode applies to launches with `--local`, or with `--config <file>` when the file has a `routing` section. A plain `rayline run claude` uses cloud mode, even while a local process is running.

`rayline run <harness> --local` needs `~/.config/rayline/settings.json` with a `routing` section, with one exception: while a local process started by `rayline local start --router local` without a `routing` section is running, `rayline run <harness> --local` uses the same defaults (Rayline CLI 0.5.4 or later). A file with only `routing` is a complete config: with no `endpoints` or `models`, every model ARC picks runs on Rayline Platform.

Claude Code also needs `routing.classifier_model`. If it's missing, `rayline local start` still starts, and ends with a note that shows your `routing` section with `classifier_model` added, for you to paste into `settings.json`. `rayline run claude --local` refuses to run and shows the same section.

When a local process is running, a cloud-mode `rayline run <harness>` (no `--local`, and no `--config` file with a `routing` section) still uses Rayline Platform and prints `Note: a local process is running; this run uses Rayline Platform. Add --local to use the local process.`

To run pi in local mode (`rayline run pi --local`), Rayline CLI needs pi 0.87.1 or later and refuses an older pi, which doesn't send its session ID. To upgrade, run `npm install -g @earendil-works/pi-coding-agent`. Current pi releases need Node.js 22.19 or later.

## Local routing decisions (preview)

By default, ARC on Rayline Platform makes each routing decision. With the local router, routing decisions run locally: ARC runs inside the local process on your machine and picks each turn's model there. The model it picks still runs where it is served, on your inference server or, for a cloud model, on Rayline Platform.

| | Who picks each turn's model | Where the turn runs |
| - | - | - |
| Cloud mode (`rayline run claude`) | ARC on Rayline Platform | Rayline Platform |
| Local mode with the hosted router (`rayline run claude --local`) | ARC on Rayline Platform | Your inference server or Rayline Platform |
| Local mode with the local router | ARC on this machine | Your inference server or Rayline Platform |

Only the routing decision is local. Turns that ARC sends to cloud models, and Claude Code's auto-mode classifier calls, still go to Rayline Platform, which is why you need to be signed in with `rayline auth login`.

Local routing decisions are a preview on Apple Silicon Macs, and on Linux and Windows with an NVIDIA GPU. They need Rayline CLI 0.5.2 or later on macOS, 0.5.3 or later on Linux, and 0.5.4 or later on Windows.

### Requirements

| Platform | Needs |
| - | - |
| **macOS** | Apple Silicon, macOS 13 or later. No other software. |
| **Linux** | x86\_64 (with AVX2) or arm64, glibc 2.28 or later, and an NVIDIA GPU with the NVIDIA driver. No CUDA toolkit. |
| **Windows** | x86\_64, Windows 10 version 1903 or later or Windows Server 2022, and an NVIDIA GPU with NVIDIA driver 576 or later. No CUDA toolkit or Visual C++ runtime. |

Supported NVIDIA GPUs:

* **x86\_64 (Linux and Windows):** compute capability 7.5, 8.x, 9.0 or 12.0, such as T4, A10, A100, L4, H100, and the RTX 30, 40 and 50 series.
* **Linux arm64:** compute capability 9.0 or 12.1 (GH200, GB10).
* Compute capability 10.x (B200, GB200) isn't supported yet.
* **Linux driver:** 525.60.13 or later, 570 or later for the RTX 50 series, and 580 or later for GB10.

Intel Macs, Windows on Arm, other GPUs (Vulkan), and machines without a supported GPU can't run the local router. Use local mode with the hosted router, the default, instead. Check your GPU and driver with `nvidia-smi` before you start.

`rayline local start` checks the platform, and on Linux and Windows the NVIDIA driver, before it downloads anything. Without them it stops with an error that names the problem and says to use the hosted router. If the driver is installed but ARC finds no GPU it can use, such as a GPU or driver that is too old, start stops after the download with:

```text theme={"system"}
the ARC encoder found no GPU it can use and would run on the CPU (…); local decisions need Apple Silicon, or an NVIDIA GPU with a current driver (Linux 525.60.13 or later, Windows 576 or later); set `routing.router` to `hosted`
```

To use the hosted router, set `routing.router` to `hosted` (or remove it) in a `routing` section with `decisions_model` and `policy.preset`. The hosted router has no built-in defaults, so `--router hosted` alone isn't enough when `settings.json` has no `routing` section.

### Turn it on

Set `routing.router` to `local` in `~/.config/rayline/settings.json`. The example below shows only the `routing` section; add or update it without replacing your existing `endpoints`, `models`, or other settings:

```json theme={"system"}
{
  "routing": {
    "router": "local",
    "decisions_model": "arc-swift-1.0",
    "classifier_model": "anthropic/claude-haiku-5.5",
    "policy": { "preset": "arc-policy.prefer-local.v1" }
  }
}
```

| Key | Meaning |
| - | - |
| `routing.router` | `hosted` (the default): ARC on Rayline Platform decides. `local`: ARC decides on this machine. The `RAYLINE_ROUTING_ROUTER` environment variable and the `--router` flag override it. |
| `routing.decisions_model` | The ARC product that decides: `arc-swift-1.0` or `arc-sage-1.0`. The local router refuses `arc-1.0`. |
| `routing.classifier_model` | The model that serves Claude Code's auto-mode safety classifier calls. Needed only to run Claude Code. |

Then sign in and run your tool in local mode. If a local process is already running, close the tools using it and run `rayline local stop` first, because a running process keeps the router it started with.

```bash theme={"system"}
rayline auth login
rayline local stop
rayline run claude --local
```

### First start

Installing Rayline CLI doesn't download anything for the local router. The first start with the local router downloads two things and shows their progress (one updating line in a terminal):

| Download | Size | Where it goes |
| - | - | - |
| The ARC release and its weights | About 1.1 GB | The Hugging Face cache (`~/.cache/huggingface/hub` unless `HF_HOME` or `HF_HUB_CACHE` says otherwise) |
| The runtime that runs ARC on your GPU (the ARC encoder) | About 11 MB on macOS, 0.7 GB on Linux arm64, and 1 GB on Linux x86\_64 and Windows | `~/.rayline/arc/<version>/` |

No Hugging Face login is needed. Rayline CLI checks every file against its pinned checksum or the release's signed checksums. Later starts reuse the files.

After a `rayline update`, the next start downloads the new version's encoder if `rayline update` didn't already, and removes the old version's.

#### Updating from 0.5.3 or earlier

Rayline CLI 0.5.3 and earlier installed the encoder beside `rayline`. 0.5.4 doesn't use that copy, so the first local-router start after the update downloads the encoder again. You can then delete `rayline-arc-encoder` and `rayline-arc-encoder-LICENSE` from the folder `rayline` is in (`~/.rayline/bin` by default).

### Speed

* The first decision of each new conversation reads the whole prompt: about 1 second per 10,000 prompt tokens on an Apple M-series Mac, so about 3 seconds for Claude Code's first turn. Later turns take about 0.1 seconds.
* The first one or two starts after a download can take a few seconds longer.
* On Windows, every start checks the ARC files again, so each start takes longer than on macOS or Linux.

### Limits

* **No fallback.** Every turn is routed by the local router. When it can't decide a turn, the turn fails; it is not sent to ARC on Rayline Platform instead.
* **Anthropic Messages tools only.** Claude Code, OpenCode, pi and oh-my-pi are supported. `rayline run` refuses Codex, Hermes and OpenClaw with the local router. To run them, set `routing.router` to `hosted`, run `rayline local stop`, then launch again.
* **No images or documents.** A turn that includes an image or a document fails.
* **No resumed sessions.** For Claude Code, Codex, pi and OpenCode, local mode refuses a command that resumes or forks a saved session, such as `claude --continue` or `claude --resume`.
* **Known models only.** A model in your `models` section that the ARC release doesn't know is skipped, with a warning when the local process starts.

### Billing

Local decisions are not metered. Turns served by Rayline Platform models are billed as usual. See [ARC pricing](https://docs.rayline.ai/arc/pricing.md#routes-api).

### Local routing for other tools

Other programs on your machine, such as the [pi-arc](https://github.com/rayline-ai/pi-arc) extension for your own pi, can ask the local router for each turn's model. Start the local process with the local router:

```bash theme={"system"}
rayline auth login
rayline local start --router local
```

This needs no `settings.json`. Without a `routing` section, it uses `arc-swift-1.0` and the `arc-policy.prefer-local.v1` preset, with no classifier model. While it runs, `rayline run <harness> --local` uses the same defaults for OpenCode, pi and oh-my-pi. Claude Code needs a `routing` section with `classifier_model`; start prints one for you to paste.

Find the local router with `rayline local status --json`:

```json theme={"system"}
{
  "running": true,
  "router": "local",
  "decisions_model": "arc-swift-1.0",
  "routes": {
    "url": "http://127.0.0.1:57696/v1/routes",
    "token_file": "/Users/you/.rayline/local/routes-token"
  }
}
```

The output has more fields than shown here. Send `POST` requests to `routes.url` with `Authorization: Bearer <token>`, reading the token from `token_file`. The token is never printed, and it only allows routing requests: it can't send anything to Rayline Platform. The URL and token change each time the local process starts, so run `rayline local status --json` again after a connection error or a `401`.

Each request is a [Routes API](https://docs.rayline.ai/arc/routes.md) request for `arc-swift-1.0` or `arc-sage-1.0` with the turn's `system`, `messages` and `tools` in Anthropic Messages format, plus three `routing` members:

* `routing.policy`: the routing policy preset.
* `routing.prices`: your price per million tokens for the models you offer, such as your own models at zero. Models you don't price use ARC's catalog price.
* `routing.execution`: the turn (`arc-execution-context.v2`) and the targets you offer. Each target is an ARC action, a model with a reasoning setting, named by its action ID, with `placement` `local` or `cloud`. Give a local target a `capacity` that says whether the prompt fits its context window; the prefer-local preset leaves out a local target whose capacity is unknown.

This request offers GLM-5.3-Flash on your own server, priced at zero, and two Claude models on Rayline Platform for a coding agent's first turn. Save it as `route-request.json`:

```json theme={"system"}
{
  "model": "arc-swift-1.0",
  "max_tokens": 32000,
  "system": "You are a coding agent working in the user's repository.",
  "messages": [
    {
      "role": "user",
      "content": "Rename the function parse_cfg to parse_config everywhere in this repo and run the tests."
    }
  ],
  "tools": [
    {
      "name": "bash",
      "description": "Run a shell command.",
      "input_schema": {
        "type": "object",
        "properties": {
          "command": {
            "type": "string"
          }
        },
        "required": [
          "command"
        ]
      }
    }
  ],
  "routing": {
    "policy": {
      "preset": "arc-policy.prefer-local.v1"
    },
    "prices": {
      "z-ai/glm-5.3-flash": {
        "input": 0,
        "cache_read": 0,
        "cache_write": 0,
        "output": 0
      }
    },
    "execution": {
      "version": "arc-execution-context.v2",
      "turn": {
        "id": "turn-1",
        "attempt": 1,
        "epoch": "epoch-1",
        "retry_of": null,
        "reconsult": null
      },
      "event": {
        "kind": "task_start",
        "basis": "explicit_client",
        "compaction": null
      },
      "last_served": null,
      "history": [],
      "hold": null,
      "expect": null,
      "targets": [
        {
          "action_id": "bd202891c3ed5bf9af5250868da17cb3ad68a05184ec5fa757876880257a5b26",
          "worker": "glm-5.3-flash",
          "placement": "local",
          "cache": null,
          "capacity": {
            "fits": true,
            "reason": null,
            "required": 33200,
            "available": 1000000,
            "available_source": "configured",
            "units": null,
            "provenance": "caller_bounded"
          }
        },
        {
          "action_id": "e8125a6d6e00df0bafbaf85796c433241de68d73b59af55c02c6a06ceae0bec2",
          "worker": "claude-haiku-5.5",
          "placement": "cloud",
          "cache": null,
          "capacity": null
        },
        {
          "action_id": "160b92a72c09743f353721b9fc50c13d655e2f231bda060d629e37ccc059851f",
          "worker": "claude-opus-5.5",
          "placement": "cloud",
          "cache": null,
          "capacity": null
        }
      ]
    }
  }
}
```

Send it with the token and URL from `rayline local status --json` (this uses `jq`):

```bash theme={"system"}
STATUS=$(rayline local status --json)
TOKEN=$(cat "$(echo "$STATUS" | jq -r .routes.token_file)")
curl -sS "$(echo "$STATUS" | jq -r .routes.url)" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -H "X-Rayline-Wire-Format: anthropic.messages.v1" \
  -H "X-Conversation-Id: conv-1" \
  -H "X-Rayline-Branch-Id: main" \
  --data @route-request.json
```

The response, trimmed:

```json theme={"system"}
{
  "object": "route",
  "route_id": "rt_<route id>",
  "model": "z-ai/glm-5.3-flash",
  "worker": "glm-5.3-flash",
  "thinking": {
    "budget_tokens": null,
    "mode": "on"
  },
  "presets": {
    "swift": {
      "model": "z-ai/glm-5.3-flash",
      "identity": {
        "action_id": "bd202891c3ed5bf9af5250868da17cb3ad68a05184ec5fa757876880257a5b26"
      }
    }
  },
  "alternatives": [
    {
      "model": "anthropic/claude-opus-5.5",
      "score": 0.98,
      "worker": "claude-opus-5.5"
    },
    {
      "model": "anthropic/claude-haiku-5.5",
      "score": 0.95,
      "worker": "claude-haiku-5.5"
    }
  ],
  "receipt": {
    "policy": "arc-policy.prefer-local.v1",
    "decided_by": {
      "model": "prefer_local_threshold",
      "thinking": "schedule"
    },
    "excluded": []
  },
  "routing": {
    "execution": {
      "episode_identity": "session_branch",
      "status": "applied",
      "turn": {
        "attempt": 1,
        "epoch": "epoch-1",
        "id": "turn-1",
        "reconsult": null,
        "retry_of": null
      },
      "version": "arc-execution-context.v2"
    },
    "policy": {
      "preset": "arc-policy.prefer-local.v1",
      "status": "applied"
    },
    "prices": {
      "models": {
        "z-ai/glm-5.3-flash": {
          "cache_read": 0.0,
          "cache_write": 0.0,
          "input": 0.0,
          "output": 0.0,
          "source": {
            "cache_read": "caller",
            "cache_write": "caller",
            "input": "caller",
            "output": "caller"
          }
        }
      },
      "price_sha256": "2b4f8fde77652829c4980a45c5bdcd2a65da72644010cec9e37663731ef9b5cf",
      "status": "applied",
      "unused": []
    },
    "unapplied_keys": []
  },
  "usage": {
    "cache_read_tokens": 0,
    "encoded_input_tokens": 130
  },
  "latency_ms": 9
}
```

Use the route only when `routing.execution.status` and `routing.policy.status` are `applied`. `model` is the model to run the turn on, and `presets.swift.identity.action_id` (`presets.sage` for `arc-sage-1.0`) is the action ARC chose. `routing.prices.models` lists each offered model's price and its `source`: `caller` for yours, `catalog` for ARC's.

A model the ARC release doesn't know is never chosen:

* An offered target whose action the release doesn't know is left out and listed in `receipt.excluded` with the reason `unknown_action`. The request still gets a route.
* A priced model that no offered action runs is listed in `routing.prices.unused`.
* If no offered action is one the release knows, there is nothing to pick: the request is refused with `400` and the code `no_feasible_target`, with no receipt.
* An unknown action named in `routing.execution.history`, `last_served` or `hold` is refused with `400` and the code `execution_unknown_action`.

[pi-arc's Rayline API notes](https://github.com/rayline-ai/pi-arc/blob/main/docs/rayline-api.md) describe how it fills each field.

There is no fallback here either: when the local process isn't running or can't decide, the turn fails. pi-arc does all of this for you.

## Operational checks

Check available memory, endpoint health, and the configured model before starting a large session. Local inference has hardware and electricity costs even when there is no hosted token charge.

Rayline CLI's local configuration controls the destinations on your device. See [connection options](https://docs.rayline.ai/rayline-cli/connections.md) for how it connects to local models and cloud services.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.