Skip to content

Consolidated Index

One unified lexical+dense hybrid search substrate (work_buddy/index/) serving every work-buddy corpus — knowledge units, vault chunks, conversation spans, Chrome tabs, summaries, projects, task notes — from a single SQLite DB. Composable ports (Partition/Encoder/ResidentCache), one shared store keyed by a partition column, builds serialized on a DB-wide writer gate, FTS5⊕dense RRF search with metadata filter pushdown. Flag-gated by index.enabled; each consumer routes via its own index.consumers. gate.

Details

Overview

One unified search substrate (work_buddy/index/) serving lexical⊕dense hybrid search over every work-buddy corpus from a single SQLite DB. It generalizes the per-domain index patterns — whole-unit knowledge ranking, the IR engine's multi-source Document/Projection model, and the vault index's warm-resident SQLite+FTS5 serving — into one composable engine.

Flag-gated: inert until index.enabled is true (default false). It builds into its own separate DB and nothing routes to it until a consumer's gate is flipped, so it can be present and kept fresh without changing live search behavior.

Object model (composable ports)

  • Partition (partition.py) — the source PORT (a Protocol + lazy factory registry). A domain adapter that discover()s indexable items (with a change-detection signal) and parse()s each into one or more Documents. Seven are registered: knowledge, vault, conversation, projects, chrome, summary, task_note (knowledge → KnowledgePartition, vault → VaultChunkPartition, the five IR sources → IRSourcePartition).
  • Document (model.py) — fields (→ BM25/FTS), projections (→ dense; scalar or pooled list), metadata (JSON, filterable), display_text, content_hash (change detection), timestamp (recency).
  • Encoder (encode.py) — kind-aware dense encode through the embedding service (label → symmetric leaf-mt; passage → asymmetric leaf-ir query/document pair).
  • ResidentCache (resident.py) — warm per-(partition, projection) dense matrices for fast serving, version-invalidated on each build.

Store

One single-writer SQLite DB (db/index-consolidated.db) holds all partitions in shared tables keyed by a partition column: documents, doc_fts (standalone FTS5 over title/body/tags), doc_vectors (float16 blobs, FK-cascaded to documents), indexed_items (the mtime/content-hash change ledger), and index_meta (KV — per-partition build_version). A fresh connection per operation (WAL, foreign keys on). Because one DB has many would-be writers across processes, writes use a busy timeout plus backoff-retry (see dev notes).

Build

IndexBuilder (build.py) runs an incremental, resumable build of one partition: discover → diff by change-key (content-hash default, or mtime) → for each changed item delete its old docs, parse, upsert, encode projections → prune deleted items → bump build_version and invalidate the partition's resident matrices. It runs under a DB-wide writer gate + the per-partition lock, so concurrent builds (across partitions or processes) serialize safely rather than colliding on the shared DB. Default is incremental (force=false).

HybridSearcher (search.py) fuses three signals: FTS5 bm25() (title/body/tags column weights, per-partition via fts_weights) ⊕ per-projection dense cosine (over the resident matrix) ⊕ Reciprocal Rank Fusion (per-partition rrf_k). Metadata filters are pushed down (filter-then-rank, so a tight filter still returns a full top-N), with optional recency bias and an optional per-source diversity cap (max_per_source). UnifiedIndex.search / search_many federate a query across partitions.

Config + flags

index:
  enabled: false                   # master kill-switch — whole index inert when false
  consumers:                       # per-consumer routing gates; a consumer routes here
    agent_docs: false              #   only when index.enabled AND its gate are true
  partitions:
    <name>:
      rrf_k: 20                    # per-partition fusion constant
      recency: false               # recency bias on/off
      coverage: active             # "active" (working set) | "all" (incl. archived/closed)
      fts_weights: null            # FTS5 bm25 (title, body, tags) column weights; null = default (3,1,2)
      max_per_source: null         # cap top-k hits per source document (diversity); null = uncapped

coverage selects how much of a source's history a partition indexes; query-time Query.filters then narrow within it (so one corpus serves both retrospective and live-work queries). See HISTORY-PARTITION-COVERAGE design notes.

fts_weights overrides the default title-leaning FTS column weights for a partition whose title field is not its most relevant text — e.g. vault, where the "title" is a navigational heading breadcrumb, so it ranks body content above it. max_per_source caps how many top-k hits may share one source document so a chunk-heavy file can't flood results; it is score-guarded, so a genuinely dominant document with no competitive alternative keeps its slots.

Capabilities, crons, endpoints

  • index_rebuild (context/index-rebuild) — incremental per-partition build; self-skips while any index build is running.
  • Five index-<partition>-refresh sidecar crons keep the active partitions fresh at corpus-matched cadences (knowledge/chrome */15, summary */30, conversation hourly, vault every 6h); one job per partition by design.
  • Embedding-service endpoints /index/search, /index/search_many, /index/build, with index_search / index_search_many clients (see architecture/embedding-service).
  • Status/dashboard seam — registered as the consolidated index in work_buddy/indexing/ (the index-agnostic status + bulk-build adapter).

Relationship to the per-domain indexes

The partitions wrap the same sources the legacy indexes serve — KnowledgePartition over the knowledge store, IRSourcePartition over the IR sources, VaultChunkPartition over the vault chunker. Each consumer is re-pointed onto the consolidated partition behind its own flag and validated (blind A/B) before the corresponding legacy index is retired. See architecture/vault-index, architecture/knowledge-system, and context/index-rebuild.