Vault Semantic Index

Native chunk-level semantic search over configured Markdown roots ("vaults"), in work-buddy's own processes (not Obsidian's heap). Heading-aware chunker → SQLite chunk store with float16 vector blobs + FTS5 → hybrid lexical⊕dense (RRF) search served warm from the embedding service. Reached via the vault_search / vault_index / vault_config capabilities and a 5-minute incremental cron.

Details

Overview

Chunk-level semantic search over one or more configured Markdown roots ("vaults" — any indexed directory, Obsidian or not). Runs in work-buddy's own processes: a heading-aware chunker, a disk-backed SQLite store, and hybrid lexical⊕dense retrieval. The index DB is a derived, disposable cache — the Markdown files are the source of truth, so a lost DB is rebuilt, never recovered.

Pipeline

FilesystemSource.discover()   multi-root walk; dot-dir skip; include/exclude globs; mtime staleness
  → MarkdownHandler (chunker)  heading-structural, code-fence aware; breadcrumb-prefixed embed_input;
                               max-size paragraph-first splitter; non-overlapping leaf sections
  → store (SQLite)             chunks + float16 vector blobs + FTS5 lexical index + indexed_items
  → embedding service          dense encode (leaf-ir document model), optional LM Studio peer offload
  → hybrid search              FTS5 bm25 ⊕ dense cosine, RRF-fused

A content-handler registry keys handlers by file extension (Markdown-only by default); an unclaimed extension is skipped. Chunk is the stable contract between handlers and the rest.

Search

vault_index.search.search(query, *, top_k, method, vault_id, recency) returns IR-compatible result dicts (doc_id, score, bm25_score, dense_score, source="vault_index", display_text, metadata). method ∈ hybrid (default) / lexical / dense. Lexical is SQLite FTS5 (bm25(), disk-backed — no in-RAM corpus); dense is cosine over the resident vector matrix; the two are fused with Reciprocal Rank Fusion. Graceful degradation: if the embedding service is unavailable the query encoder returns None and search falls back to lexical-only — never an error.

Capabilities + cron

• vault_search — hybrid search → markdown results (degrades to in-process lexical if the service is down).
• vault_index — status (per-vault counts, read locally so it works with the service down) / build (incremental reconcile + encode, skips if a build holds the lock).
• vault_config — add/update/remove a vault in vault_index.vaults (writes config.local.yaml).
• A sidecar cron (sidecar_jobs/vault-index-rebuild.md) calls vault_index build every 5 minutes; incremental, fast when nothing changed.

Configuration (multi-vault)

vault_index.vaults is a mapping keyed by stable id: {id: {path, include, exclude}}. Empty (the default) synthesizes a single vault from vault_root. The id is the stable identity (survives a path change) and namespaces chunk ids. Editing config takes effect on the next build — no restart.

Dashboard

Settings › Embeddings shows a read-only System table (IR + knowledge indexes) and an editable Your vaults table (per-vault counts + a gear to edit path/globs, add/remove). Backed by /api/embeddings over the index-agnostic seam.

Key files

• work_buddy/vault_index/{chunker,handlers,source,store,indexer,dense,dense_cache,search,status}.py
• work_buddy/vault_index/__main__.py — manual/dev build CLI.
• work_buddy/indexing/ — the index-agnostic status seam (IR / vault / knowledge adapters).
• work_buddy/mcp_server/ops/vault_ops.py — the three capability dispatchers.
• work_buddy/embedding/service.py — the in-service /vault/search + /vault/index host.

See architecture/embedding-service for the dense-encode backend and architecture/inference/broker for the BACKGROUND-priority admission control the bulk encode rides on.