Sidecar Daemon

Unified process supervisor, cron scheduler, and message-driven job dispatcher

Details

A single long-lived Python process that replaces multiple independent Windows Task Scheduler entries with a unified process supervisor, cron/heartbeat scheduler, and message-driven job dispatcher.

Starting: uv run python -m work_buddy.sidecar

Single-instance is enforced by an OS lock on <data_root>/runtime/sidecar.lock, held for the daemon's lifetime and released by the kernel on death. The PID file (<data_root>/runtime/sidecar.pid), its process-identity companion (sidecar.pid.identity.json) and the state file (<data_root>/runtime/sidecar_state.json) record which daemon is running; they do not gate whether a second one may start.

Two loops split the daemon's work by blocking behavior. The supervisor loop (main thread) evaluates cached health probes, restarts failed children, and writes sidecar_state.json every tick — everything on it is fast and bounded, so the state file's freshness is a true daemon-liveness signal (wbuddy status classifies ~90s of staleness as wedged). The dispatch loop (a background thread) runs scheduler cron ticks, message-driven dispatch, and retry sweeps — the phases that execute jobs and replays inline and can legitimately block for minutes (agent spawns, index rebuilds, local-LLM leases). A slow job therefore reads as a busy dispatch phase in the state file, never as a hung daemon, and can never delay child restarts.

Three subsystems: 1. Process Supervisor — starts and monitors child services. A dedicated HealthMonitor thread probes every service's /health endpoint concurrently (via ThreadPoolExecutor) at health_probe_interval (default 5s), so dispatch work or slow skills can never delay probes. The supervisor loop consumes cached health state only; a service is restarted after health_failure_threshold (default 2) consecutive failed probes, under exponential backoff, and given up on after max_service_crashes (default 5). 2. Cron/Heartbeat Scheduler — loads job .md files from BOTH sidecar_jobs/ (system jobs, git-tracked) AND <data_root>/user_jobs/ (user-authored, gitignored) and fires them on their cron schedule, inline on the dispatch loop. Supports exclusion windows (quiet hours), and optional per-job stable jitter to spread phase-aligned starts. Hot-reload has two triggers: a JobsWatcher (kernel filesystem events via watchdog; ~50ms latency on file change) AND a 30s polling interval as a safety net. Watcher events set a threading.Event (Scheduler.jobs_reload_pending) that the daemon's dispatch-loop sleep waits on, so the next cycle reloads immediately. On filename-stem collision the user file wins and a WARN is logged. 3. Message-to-Job Dispatch — polls messaging service for pending messages addressed to work-buddy. Classifies each message and executes automatically, also inline on the dispatch loop.

Shutdown: First Ctrl+C / SIGTERM requests graceful shutdown; a watchdog thread force-exits after 15s if the main thread is stuck in a blocking syscall, so shutdown is always bounded. A second Ctrl+C force-kills children immediately. The JobsWatcher observer thread is stopped and joined alongside the HealthMonitor in the cleanup path; the dispatch thread is a daemon thread, briefly joined and abandoned if mid-job. The pid file is removed on shutdown only when it still records the exiting process's own pid — the atexit hook can fire after a takeover has replaced the file, and deleting the successor's pid file would make a healthy daemon read as not running.

Child stdout/stderr: redirected to <data_root>/runtime/service_logs/<service>.log, after a start banner, so a silent or crashing child is always observable. start_host opens the log, hands it to the child, and closes its own copy. At child launch the daemon rolls an oversized live log (>16 MiB) aside to a timestamped backup (_roll_oversize_log), and the service-logs artifact then reaps rolled backups older than 7 days on the twice-daily cleanup tick (pinning the live log). See the rotation dev-note above.

Job file format: .md files in either jobs directory with YAML frontmatter (schedule, recurring, type, skill/params, enabled, spawn_mode, optional jitter_seconds). Each loaded Job carries a source field ("system" or "user") that propagates through JobState into sidecar_state.json and is used by the dashboard's Jobs tab to group entries.

Job types: skill (dispatches a canonical registered Skill), workflow (resolves a canonical Workflow name, executable alias, or stable wfd_* ID and invokes it through WorkflowService), prompt (freeform text — spawns a headless agent session, consent-gated).

Scheduled Workflow facilities and admission

The sidecar does not call the conductor directly. Its Workflow adapter constructs a headless WorkflowInvocationContext and advertises the facilities it can actually provide: program execution and subagent spawning. It does not advertise a calling-agent reasoning facility. WorkflowService applies the same registry resolution, address-first authorization, structured admission, strict param validation, and stable-ID/revision start guard used by the MCP adapter. The MCP adapter supplies the interactive consent coordinator; the sidecar instead relies on the conductor-minted run grant in its isolated headless session, with a six-hour safety ceiling. If a Step needs a missing facility—or a required preference, component, context, or mode fails—the request returns a structured denial and no run is created. Dependencies reachable only through optional Steps do not inflate the compiled required-component set.

Agent spawn modes for prompt jobs: headless_ephemeral (default, --print --no-session-persistence), headless_persistent (--print only, registered in <data_root>/runtime/agent_registry.json), interactive_persistent (deferred).

Session launcher (work_buddy/session_launcher.py): remote_session_begin launches a visible Claude Code terminal session, optionally with --remote-control. Consent-gated. Primary use case: Remote Control from phone.

Cron syntax: standard 5-field cron in the timezone set in config.yaml (default: America/New_York).

Jitter — the thundering herd problem and how the scheduler avoids it

Many of work-buddy's jobs run on phase-aligned schedules: */3, */5, */10, */30. These coincide at common minute boundaries (:00, :30, hourly, etc.) and fire simultaneously — every five minutes a wave of indexers, sync jobs, and health checks all hit at the same second. That's the thundering herd: a synchronized burst of work whose contention (CPU, disk, lock acquisition, downstream API rate limits) is much worse than the same total work spread across the interval. Each individual job is fine; the simultaneous-start is the problem.

The scheduler's per-job jitter solves this: set jitter_seconds: <N> in a job's frontmatter and it fires at scheduled_at + offset where offset is in [0, N], deterministic per job. The same job lands at the same offset every cycle; two jobs sharing a schedule land at different offsets and stop colliding.

The dashboard form caps jitter_seconds per schedule. Worked examples: */3→10s, */5→30s, */10→60s, */30→180s, hourly/daily/weekly→300s (the hard cap is 5 minutes regardless). The cap is a UI recommendation, not a backend rule — hand-editing a .md file can exceed it.

Tick cadence is health_check_interval (default 30 s), so values < 30 are quantized away in practice — the dashboard form warns when the typed value falls below that floor. jitter_seconds: 0 (the default) bypasses the pending-fire queue entirely and fires inline on cron match.

Observability fields next_at (raw cron eligibility) and effective_at (next_at + offset, or queued pending due time) ride alongside each JobState in sidecar_state.json; the dashboard's Jobs tab renders effective_at in the "Next Run" column and shows the configured jitter_seconds in a dedicated "Jitter" column so users can correlate display time with cause.

Config (sidecar: section in config.yaml): health_check_interval (cadence of both loops: supervisor restart-decision evaluation and dispatch cycles), health_probe_interval (HealthMonitor cadence), health_probe_timeout, health_failure_threshold, max_service_crashes, restart_backoff_base, dispatch_stall_warn_seconds (default 600 — warn when the dispatch loop sits in one phase this long), services (with module/port/enabled per service), jobs_dir (system jobs, defaults to sidecar_jobs), user_jobs_dir (user jobs override; empty = <data_root>/user_jobs/), heartbeat, message_poll_interval.

Observability: The supervisor writes <data_root>/runtime/sidecar_state.json every tick regardless of what the dispatch loop is doing. Alongside services/jobs/events, the state carries dispatch-loop fields: dispatch_phase (scheduler_tick | message_poll | retry_sweep | idle), dispatch_phase_since, dispatch_job (the job currently executing inline, if any), and last_dispatch_at (end of the most recent dispatch cycle). A phase held past dispatch_stall_warn_seconds emits a dispatch_stalled event (once per stall) naming the phase and job; wbuddy status prints a busy line for a phase held past ~2 minutes. Query via sidecar_status or sidecar_jobs skills. Per-service child logs at <data_root>/runtime/service_logs/*.log (rotated as described above). Dashboard subscribes to cron.hot_reload events on the bus to refresh its Jobs tab on every actual reload.

Every reader of the state file goes through sidecar.state.read_state_json. The daemon replaces the file atomically several times a minute, and on Windows a read that lands during a replacement fails with a permission error. The reader retries such a read, so a rewrite in progress is never mistaken for a missing or wedged daemon.

Runtime context

The daemon and its services are hosts in the sense of architecture/process-execution. The first statement of python -m work_buddy.sidecar is establish_host_context(HostRole.SIDECAR), which gives a daemon started under pythonw.exe (the logon task, or a hand-made shortcut) a console with no window, and gives every daemon a fresh sidecar- session id. wbuddy start, provisioning, the tray and the desktop launcher start it through start_host(HostRole.SIDECAR, ...), which starts python.exe with CREATE_NO_WINDOW. Services start through start_host(HostRole.SERVICE, ...). Either way, the console programs the daemon and its services run inherit a console with no window. Every launch the daemon's own code makes goes through work_buddy.process.

At boot the daemon records the context it actually has in the host field of the state file. wbuddy status prints it as a Runtime: line and warns on drift, and the sidecar health component's runtime-context step checks the same record.