Installation and Deployment
The cross-platform kanade deploy design and script retirement gates are in SPEC §2.13. Typed schema/validation and plan persistence are implemented, but production typed dispatch and the deploy CLI are unavailable; keep using the current procedures below during migration. It targets enrolled agents only and leaves the Agent Installation screen one-liners unchanged.
This section details how to bootstrap kanade components as native Windows services in production or staging environments.
Deployment Model
Production hosts and target endpoints run kanade components as background Windows services, providing automatic startup and configured service-manager recovery.
| Service Name | Triple / Binary | Config Source | Typical Target |
|---|---|---|---|
| KanadeNats | nats-server.exe | Config file with token substitution / ACLs | Central server |
| KanadeBackend | kanade-backend.exe | Hardened Registry / Config file | Central server |
| KanadeAgent | kanade-agent.exe | Config file / hardened Registry / local state DB | Managed endpoints |
1. Prerequisites
- Host OS: Windows 10/11 or Windows Server 2016+.
- gsudo: Required to perform elevated installations from standard user shells (or run commands from an Administrator-level PowerShell prompt).
- Network Routing: Managed endpoints must be able to reach the NATS server port (default
4222) over TCP.
2. Setting Up the NATS Server (Broker)
The NATS server acts as the messaging core.
- Stage the deployment bundle using
scripts/build-release.ps1 -Roles nats. - Deploy the service with elevation:
# Elevated PowerShell prompt & "dist\nats\deploy-nats.ps1" -NatsToken "your-secure-nats-token" -Recreate
This installs the KanadeNats service, configures it to run under the local system account, sets up JetStream data directories, and substitutes the token into the installed NATS configuration file, and restricts that file to SYSTEM and Administrators.
Optional: per-role users instead of the shared token
By default the broker runs on the single shared token above. To run the three
role users (agent, backend, breakglass) instead, deliberately:
# Default cost 8 requires a long random password (32 random alphanumeric characters recommended).
# Hashes only, minted with scripts/ops/nats-password-hash.ps1 (or .sh); never plaintext.
$env:KANADE_NATS_AGENT_PASSWORD_HASH = '<hash>'
$env:KANADE_NATS_BACKEND_PASSWORD_HASH = '<hash>'
$env:KANADE_NATS_BREAKGLASS_PASSWORD_HASH = '<hash>'
& "dist\nats\deploy-nats.ps1" -UseNatsUsers # switch
& "dist\nats\deploy-nats.ps1" -UseNatsToken -NatsToken "your-secure-nats-token" # revert
-UseNatsUsers replaces the authorization block of the installed config with
an include of nats-server.users.conf (the template with the hashes
substituted) and gives both files the SYSTEM + Administrators ACL;
-UseNatsToken restores the shipped token block (hand edits inside it are not
restored) and removes the users file. The service is stopped and started by
the script as always (-NoStart defers the start). On Linux the equivalent is
KANADE_NATS_AUTH_MODE=users|token for setup.sh, recorded so re-runs keep it,
with a manual systemctl restart nats-server to apply it (see
deploy/linux/README.md).
For the recommended bcrypt cost 8 and broker burst measurements, see Choosing the bcrypt cost.
The switch is atomic: a config cannot hold both a token and users, and a client presenting a token is rejected once users exist. Every agent, backend and CLI host must already hold a user pair before the broker flips; follow the separately documented readiness procedure first.
Expected authentication mode and connection findings
The backend watches how every connection to the broker authenticated and raises a warning when that is not what you intended, so that a switch to per-role NATS users that failed or silently reverted, or a broker that authenticates nobody, does not go unnoticed. The broker's own report is the source (its monitoring endpoint, polled every minute), not anything an agent says about itself.
Setting. Settings → Server → Expected NATS authentication (the
nats_auth_mode server setting) is either token (the default, today's
shared token) or users (per-role users). It is an expectation only: it
changes nothing about how any process connects, it just defines what counts as
unexpected. Set it to users when you switch the broker to the users block,
and back to token if you switch back.
Findings. Each connection is judged by the mode you expect, the role it
announces in its connection name (kanade-<role>[/identity]) and the user the
broker reports it authenticated as:
| finding | when | severity |
|---|---|---|
broker_open | positive broker evidence of access without an authenticated identity (no-auth), in either mode | critical |
token_reverted | users expected, but the connection used the shared token | critical |
role_user_mismatch | users expected, but the user is not the announced role's (agent, backend, breakglass for the CLI), e.g. the backend user on an agent connection | warning |
credential_unnameable | users expected, but the credential could not be named | info |
Under token, the shared token is expected and only broker_open is
reported. The announced role is chosen by the client, so a mismatch is
evidence of a misconfigured or misused host, while a match proves nothing: it
is only the connection's claim, compared against the broker's word.
Findings are grouped by kind and announced role, so a whole fleet reverting at
once is a handful of entries, each with a connection count and at most five
registered host names. A finding is raised when it first appears, shows as a
banner on the Dashboard, and is marked resolved when it is gone (a later
recurrence is raised again). While the backend's own broker connection
cannot prove the mode, user names read as unknown, so expect
credential_unnameable entries until it reconnects.
Acceptance evidence and uncertainty. A TCP connection is already visible in
/connz?auth=1 after INFO, before CONNECT or successful authentication. A
server-issued cid, an empty/missing authorized_user, a client name, language,
version, RTT, or elapsed time alone does not prove access without authentication.
The projector brackets its paged connz walk with /varz, checking server id,
startup/configuration timestamps and response/page metadata. A restart, reload,
identity change, malformed/missing response or timeout invalidates the poll.
Pages are sorted by ascending CID and overlap at the preceding boundary CID.
New connections append and do not by themselves prevent resolution; a missing
boundary, duplicate/reordered CID or incomplete walk holds resolutions, while
usable positive observations can still raise findings. The cumulative
total_connections counter is not a completeness condition. Offset pagination
is not an atomic snapshot: the boundary checks detect skipped prefixes, not
every possible race or a change immediately after the final response.
With no reported identity on TCP, application message traffic (in/out messages)
or a live subscription proves the client was allowed to use the protocol.
Positive traffic must also be consistent with broker policy: explicit
auth_required: true plus a missing identity stays unproven. Counters can predate
a second CONNECT, whose options are overwritten before re-authentication.
All three
counters and the client kind/transport must be present; missing is not zero.
An explicit TCP auth_required: false also proves that listener is open. The
normally omitted false field is conservatively unknown, not inferred false:
a silent open connection can remain unproven until application activity is seen.
The backend normally has subscriptions, so an active open broker remains visible
without waiting for a grace period. An idle authentication-required TCP connection
within the broker's own auth_timeout is auth-pending; after that budget, or
without enough monitoring evidence, it is auth-unproven. Expiry never proves
acceptance. These are different from unknown, which still means a reported
credential cannot safely be named and retains credential_unnameable handling.
Pending/unproven connections do not replace an agent's last observed credential.
TCP auth policy and timeouts are never applied to WebSocket connections. WS can
have separate authorization, credentials from cookies, or no_auth_user.
The current monitoring fields do not expose its effective authorization policy.
Without a reported identity, even positive traffic remains
auth-unproven with reason listener_auth_unproven: counters may predate a
second CONNECT whose replacement credentials will be rejected. This also limits
detection of truly open WS listeners; TCP acceptance evidence remains available.
A reported identity takes precedence over activity. Cookie/token auth retains
redaction. WS user values stay unknown because a live TCP user login cannot
prove that another listener's value is a username rather than a secret.
no_auth_user assigns a broker identity, so this audit cannot distinguish that
fallback from a credentialed login as the same user: it is not proof of a
fully open broker. WS authentication without an identity remains unproven.
MQTT/leaf/unknown transports without a reported identity are also unproven. This audit trusts the
configured monitoring endpoint to be the broker and to honor auth=1; it does
not test listener reachability or actively probe production with empty credentials.
Any pending/unproven observation holds existing finding resolutions; it does not
block a new finding proven by another connection. Health reports
authentication_uncertain, with pending_connections and unproven_connections
counts (including partial-poll observations for incomplete). The optional
uncertainty_reasons map breaks those observations down using fixed, secret-free
EvidenceReason codes: pre_connect_pending, incomplete_monitoring,
listener_auth_unproven, invalid_connection_start, auth_policy_conflict, and
acceptance_unproven. Codes describe evidence limitations, not a proven cause.
The notice ledger's hold_resolve_kinds policy preserves existing notices while
allowing positive findings from other connections, including new critical alerts.
Counts and reason maps are null
when unavailable or stale. The aggregate metadata uses a JSON envelope in the
existing audit-state TEXT field; legacy plain statuses remain readable.
Connection evidence uses the existing retention ledger described below;
Observed.connection retains broker identifiers and timestamps alongside the
classified label. auth-pending/auth-unproven never contribute finding evidence.
External consumers should use the health API rather than interpret that internal
field.
The unnamed critical observed on 2026-10-18 from 02:24:58.636897559 UTC to 02:25:58.230333505 UTC after an OOM-triggered broker restart is a likely false positive. The pre-CONNECT window is reproducible on authentication-required NATS 2.14.3, but retained historical connection evidence cannot establish that this was the exact cause of that incident.
The evidence rules follow NATS 2.14.3's monitoring implementation and monitoring documentation. Its protocol parser gates application operations while awaiting authentication; its CONNECT handler sets metadata before checking authentication, and authentication implementation handles listener overrides and fallback identities.
Regression checks use fixtures, local HTTP mocks and disposable brokers only:
cargo test --locked -p kanade-backend nats_auth
cargo test --locked -p kanade-backend nats_conns
cargo test --locked -p kanade-backend live_connz -- --ignored
cargo make check
The live tests need nats-server on PATH and permission to bind loopback
sockets. They exercise INFO-before-CONNECT, rejected credentialless CONNECT,
anonymous authenticated token/user connections, reconnects, an open broker,
WS token overrides and WS no_auth_user. Run against the supported broker
version when upgrading; fixtures do not substitute for checking changed semantics.
Grace after a switch. When you change the expected mode, new findings are held back for 5 minutes. Switching re-authenticates every client, and each reconnects with backoff while the poll only looks once a minute, so for a few polls the broker's view is a mix of old and new; the window keeps those stragglers from being reported against the mode you just chose. Re-saving the same mode does not start it again, and findings already open still update and resolve during it. The Dashboard says when the window is open.
Reading it from outside. GET /api/health/fleet carries a nats_auth
block: expected_mode, grace_until, poll_status, open_total (the
connections covered) and findings[] (kind, subject, severity, count,
sample_hosts, first_seen_at, occurrence_id). It does not change the response's
status or HTTP code, so an existing monitor keeps behaving as before; a
monitor that should alert on this must read nats_auth.findings. A
poll_status other than ok means the audit could not judge the whole broker
(authentication_uncertain, incomplete, endpoint_unreadable,
settings_unreadable, state_unreadable, stale, or pending before its first
poll), and an empty list is then not a clean result.
What is never shown. The reported value of a connection may be a secret on
some broker builds, so findings carry fixed vocabularies (kind, severity,
the role bucket agent / backend / cli / unnamed / other), counts,
registered host samples and an occurrence reference. Admin-only connection
evidence adds broker-issued identifiers and parsed timestamps, as described
below. Neither the credential, nor the reported user name, nor the raw
connection name is copied into notice evidence.
Retained NATS authentication evidence
Each new occurrence has its own numeric occurrence_id in the authenticated
fleet health finding. The aggregate backend_notices row remains compatible;
reopening it does not overwrite earlier occurrences. Dashboard's admin-only
NATS authentication evidence and resolved history details and
GET /api/notices/nats-auth expose the latest 100 occurrences, including resolved
ones. Pass ?before=<oldest-id> to page backwards. Viewers and operators cannot
read this endpoint. Existing service-token and auth-disabled administrator
semantics still apply. Identifiers are selectable and copyable for log searches.
A host claim is saved only for an agent role whose exact pc_id is registered; registration does not authenticate the claim. Stored host text removes control characters and is limited to 64 Unicode characters. Other sample text is bounded by typed timestamps, fixed role/credential vocabularies, decimal u64 cids and ASCII alphanumeric server IDs of at most 64 bytes. Invalid identifiers are rejected rather than truncated. Older samples have no host field; it is not backfilled. The Dashboard separates broker identifiers and credential classes from untrusted client role/host claims.
A sample records the broker-issued decimal cid, server_id and broker
startup timestamp (/varz.start), backend first/last observation times (page
fetch completion, before settings/database work), broker page observation time, and connection start/last activity timestamps when supplied.
Occurrence timestamps describe notice processing; individual sample timestamps
describe page acquisition. A cid alone, or server_id alone, must not be treated
as a restart identity.
The collector reads /varz before and after the paginated /connz walk and
checks each page's server_id, timestamp and duplicate cids, within the existing
20-second timeout. Missing identity or an inconsistent/restarted broker rejects
the snapshot, reports an unreadable poll and leaves open findings unresolved.
A paging cap reports an incomplete poll and likewise cannot resolve findings.
This is a best-effort walk of live connections, not an atomic broker snapshot.
Evidence includes only a fixed credential class (user, shared-token,
no-auth, unknown), a fixed claimed_role bucket and an optional bounded
registered host claim. Role and host are untrusted client claims, unlike
broker-issued connection identifiers. Aggregate notifications also retain
registered host samples. Raw
authorized_user, tokens, passwords, connection names, arbitrary client metadata
and unrestricted monitoring responses are never copied into connection evidence.
Peer addresses are deliberately omitted: proxies and loopback commonly identify
an intermediary rather than the originating client. The admin boundary limits
exposure of retained connection correlation data.
Per occurrence, at most the first 10 sampled identities and 10 most recently
observed other identities are retained, deduplicated by server_id/start/cid,
within a 32 KiB JSON limit. Each sample and the latest occurrence observation
record whether their poll was complete; partial polls may undercount connections. Broker IDs are limited to 64 alphanumeric bytes;
other context is parsed timestamps or fixed vocabulary. sampled_out indicates
sample/size omissions. Evidence updates on each successful poll even if group
counts/host samples have not changed. Ledger changes, occurrence creation,
resolution and evidence updates commit in one SQLite transaction.
Samples expire 30 days after their last observation, including samples in open findings. Resolved occurrences expire 30 days after resolution and are also capped at the newest 1000. Cleanup runs when the projector starts and every poll (including failed polls); an offline backend performs cleanup when restarted. There are only the classifier's fixed kind/role buckets of open occurrences. SQLite backups must use the operator's corresponding retention/access policy; logical expiry does not securely erase old database pages or backups.
Old notices have no recoverable connection evidence; the UI labels absent or
expired evidence accordingly. The identifiers lost in the 2026-10-18 incident
cannot be reconstructed by this migration. Last observed is not a disconnect
time. Connections that open and close between minute polls can be missed, and
wiping the projection database loses these snapshots permanently. Retained
evidence uses the same authentication classification as aggregate
findings through Observed and the shared evaluate_observed adapter. Pending
and unproven connections do not contribute finding samples or resolve existing
occurrences. The additive evidence migration is named
20261018000000_nats_auth_evidence.sql.
3. Deploying the Backend API & SPA
The backend manages operator connections and processes event logs.
- Stage the backend binaries and React SPA bundle using
scripts/build-release.ps1 -Roles backend. - Deploy the service:
# Elevated PowerShell prompt & "dist\backend\deploy-backend.ps1" ` -NatsToken "your-secure-nats-token" ` -StaticToken "your-operator-spa-bearer-token" ` -ForceConfig -Recreate
-NatsToken: Connects the backend to the local NATS server securely.-StaticToken: Defines the API bearer token required for operator CLI/SPA logins.
The deployment script registers the KanadeBackend Windows service, sets the appropriate ACLs, and verifies the endpoint.
4. Installing the Agent on Target Endpoints
Install the agent on every endpoint PC that you want to manage.
- Stage the agent bundle using
scripts/build-release.ps1 -Roles agent. - Copy the contents of the
dist/agentfolder to the target PC. - On the target PC, run the installer:
# Elevated PowerShell prompt & ".\deploy-agent.ps1" -NatsToken "your-secure-nats-token" -ForceConfig -Recreate
The script:
- Places
kanade-agent.exeinto its destination directory. - Secures the configuration and NATS token in the Windows registry path (
HKLM:\SOFTWARE\Kanade\agent). - Registers and starts the KanadeAgent service.
Agent-role NATS user in generated installers
deploy-agent.ps1 also accepts -NatsUser / -NatsPassword (the shell
scripts take KANADE_NATS_USER / KANADE_NATS_PASSWORD), both or neither.
The installers the backend generates (Windows ZIP, Linux / macOS tarball)
embed such a pair when the server settings' agent_install section holds
both a user and a password, next to the token, so machines installed from now
on already carry the credential the broker may later require.
- The pair is the agent role's credential and is shared by all agents by design. The backend's own credential and the break-glass credential must never be entered in these settings.
- It is write-only, like the token: the API returns only
nats_user_set/nats_password_set, and a settings update that omits both keeps the stored pair. A settings update that sets only one of the two is rejected, so replacing the password means sending the user again. - A generated
README.txtstates whether a user pair is included, without revealing it; with no paragraph about it, none was embedded and the installer is exactly what it was before this setting existed.
Once the service is active, the agent establishes an outbound NATS connection, subscribes to command streams, and reports its online heartbeat back to the fleet backend.