Taylor Eernisse 4270603da4 feat(db): Add migrations for documents, FTS5, and embeddings ...

Three new migrations establish the search infrastructure:

- 007_documents: Creates the `documents` table as the central search
  unit. Each document is a rendered text blob derived from an issue,
  MR, or discussion. Includes `dirty_queue` table for tracking which
  entities need document regeneration after ingestion changes.

- 008_fts5: Creates FTS5 virtual table `documents_fts` with content
  sync triggers. Uses `unicode61` tokenizer with `remove_diacritics=2`
  for broad language support. Automatic insert/update/delete triggers
  keep the FTS index synchronized with the documents table.

- 009_embeddings: Creates `embeddings` table for storing vector
  chunks produced by Ollama. Uses `doc_id * 1000 + chunk_index`
  rowid encoding to support multi-chunk documents while enabling
  efficient doc-level deduplication in vector search results.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-01-30 15:45:41 -05:00

1.7 KiB

Raw Permalink Blame History

View Raw