Skip to content

Consolidated Index

One unified lexical+dense hybrid search substrate (work_buddy/index/) serving every work-buddy corpus — system knowledge, native Journal, Projects, Contracts and Personal Knowledge, vault chunks, conversation spans, Chrome tabs, summaries, and task notes — from a single SQLite DB. Composable ports (Partition/Encoder/ResidentCache), one shared store keyed by a partition column, builds serialized on a DB-wide writer gate, FTS5⊕dense RRF search with metadata filter pushdown. Database partitions use durable build-then-ack outbox delivery; sealed Markdown roots are detached from Vault discovery and stale index rows. Flag-gated by index.enabled; each consumer routes via its own index.consumers. gate.

Details

Overview

One unified search substrate (work_buddy/index/) serving lexical⊕dense hybrid search over every work-buddy corpus from a single SQLite DB. It generalizes the per-domain index patterns — whole-unit knowledge ranking, the IR engine's multi-source Document/Projection model, and the vault index's warm-resident SQLite+FTS5 serving — into one composable engine.

Flag-gated: inert until index.enabled is true (default false). It builds into its own separate DB and nothing routes to it until a consumer's gate is flipped, so it can be present and kept fresh without changing live search behavior.

Object model (composable ports)

  • Partition (partition.py) — the source PORT (a Protocol + lazy factory registry). A domain adapter that discover()s indexable items (with a change-detection signal) and parse()s each into one or more Documents. Ten are registered: knowledge, journal, projects, contracts, personal_knowledge, vault, conversation, chrome, summary, and task_note (the four cut-over domains own SQLite adapters; the remaining generic IR sources use IRSourcePartition).
  • Document (model.py) — fields (→ BM25/FTS), projections (→ dense; scalar or pooled list), metadata (JSON, filterable), display_text, content_hash (change detection), timestamp (recency).
  • Encoder (encode.py) — kind-aware dense encode through the embedding service (label → symmetric leaf-mt; passage → asymmetric leaf-ir query/document pair).
  • ResidentCache (resident.py) — warm per-(partition, projection) dense matrices for fast serving, version-invalidated on each build.

Store

One single-writer SQLite DB (db/index-consolidated.db) holds all partitions in shared tables keyed by a partition column: documents; doc_fts, an external-content FTS5 index over the canonical document row's title/body/tags; doc_vectors (float16 blobs, FK-cascaded to documents); indexed_items (the mtime/content-hash change ledger); and index_meta (KV, including schema and per-partition build generations). A fresh connection per operation (WAL, foreign keys on). Because one DB has many would-be writers across processes, writes use a busy timeout plus backoff-retry (see dev notes).

Build

IndexBuilder (build.py) runs an incremental, resumable build of one partition: discover → diff by change-key (content-hash default, or mtime) → for each changed item delete its old docs, parse, upsert, encode projections → prune deleted items → bump build_version and invalidate the partition's resident matrices. It runs under a DB-wide writer gate + the per-partition lock, so concurrent builds (across partitions or processes) serialize safely rather than colliding on the shared DB. Default is incremental (force=false).

Legacy IR-backed partitions whose source accepts a days parameter use an explicit 3650-day discovery window. The wrapper does not inherit the shorter rolling defaults used by the live IR jobs, so older conversations and summaries remain eligible for reconciliation.

For a database partition, the facade additionally snapshots its transactional search outbox, builds and verifies source/index parity, then acknowledges exactly the snapshot. The returned outbox receipt uses wb.search-outbox-delivery/v1; ready=true requires zero remaining lag and zero parity mismatches. force=true is the explicit per-partition restore/backfill path.

Incremental callers may bound changed/deleted source items and missing-vector batches. Reaching either limit returns explicit partial progress; the next run resumes from durable rows. The scheduled conversation job uses this path hourly so catch-up work yields the serial sidecar dispatch lane between slices.

HybridSearcher (search.py) fuses three signals: FTS5 bm25() (title/body/tags column weights, per-partition via fts_weights) ⊕ per-projection dense cosine (over the resident matrix) ⊕ Reciprocal Rank Fusion (per-partition rrf_k). Metadata filters are pushed down (filter-then-rank, so a tight filter still returns a full top-N), with optional recency bias and an optional per-source diversity cap (max_per_source). UnifiedIndex.search / search_many federate a query across partitions.

Config + flags

index:
  enabled: false                   # master kill-switch — whole index inert when false
  startup_prewarm: lazy            # lazy lexical-first warming; "all" eagerly warms matrices
  consumers:                       # per-consumer routing gates; a consumer routes here
    agent_docs: false              #   only when index.enabled AND its gate are true
  partitions:
    <name>:
      rrf_k: 20                    # per-partition fusion constant
      recency: false               # recency bias on/off
      coverage: active             # "active" (working set) | "all" (incl. archived/closed)
      fts_weights: null            # FTS5 bm25 (title, body, tags) column weights; null = default (3,1,2)
      max_per_source: null         # cap top-k hits per source document (diversity); null = uncapped

coverage selects how much of a source's history a partition indexes; query-time Query.filters then narrow within it (so one corpus serves both retrospective and live-work queries). See HISTORY-PARTITION-COVERAGE design notes.

fts_weights overrides the default title-leaning FTS column weights for a partition whose title field is not its most relevant text — e.g. vault, where the "title" is a navigational heading breadcrumb, so it ranks body content above it. max_per_source caps how many top-k hits may share one source document so a chunk-heavy file can't flood results; it is score-guarded, so a genuinely dominant document with no competitive alternative keeps its slots.

Skills, crons, endpoints

  • index_rebuild (context/index-rebuild) — incremental per-partition build; self-skips while any index build is running.
  • Ten index-<partition>-refresh sidecar crons keep the active partitions fresh at corpus-matched cadences, including Journal every five minutes and Projects, Contracts, Personal Knowledge, and Task Notes every fifteen minutes. One job per partition preserves replay and backfill isolation. Conversation refresh runs hourly in bounded, resumable slices so it cannot monopolize the serial dispatch lane for a full backlog drain.
  • Embedding-service endpoints /index/search, /index/search_many, /index/build, with index_search / index_search_many clients (see architecture/embedding-service).
  • Status/dashboard seam — registered as the consolidated index in work_buddy/indexing/ (the index-agnostic status + bulk-build adapter).

Relationship to the per-domain indexes

The partitions wrap the same sources the legacy indexes serve — KnowledgePartition over the knowledge store, IRSourcePartition over the IR sources, VaultChunkPartition over the vault chunker. Each consumer is re-pointed onto the consolidated partition behind its own flag and validated (blind A/B) before the corresponding legacy index is retired. See architecture/vault-index, architecture/knowledge-system, and context/index-rebuild.

The consolidated-index rowid-aligned FTS storage repair and backfill are prerequisites, not consumer cutover. Until parity and the consumer's cutover evidence pass, legacy search remains authoritative and its database and scheduled maintenance remain intact.

Authority-sealed Journal, Projects, Contracts, and Personal Knowledge Markdown roots are never ordinary Vault-search inputs. The filesystem source prunes each root before os.walk descends, rejects direct parse attempts, and the same partition reconciliation deletes any rows indexed before the seal. A pre-seal maintenance run uses only the bounded configured-domain operator; the authority-derived rule then maintains the fence after exposure.