Writes documents into the vector index in the background. A producer publishes a document to NSQ and moves on; embedder takes it off the queue, embeds its text with BAAI/bge-small-en-v1.5 on the embedding service at https://embed.site.quack-lab.dev, and writes its points into the document's Qdrant collection at https://vectors.site.quack-lab.dev.

Publishing a document

POST the document as JSON to nsqd's HTTP API, on the producer's own topic in the embedder's namespace, embedder. and the producer's name. nsqd is shared by every project, and embedder reads no topic outside that namespace.

curl -sS --fail-with-body --data-binary @document.json 'https://nsq.site.quack-lab.dev/pub?topic=embedder.taskman'

{
  "collection": "taskman-tasks",
  "id": "board:4503",
  "fields": {"source": "board", "project": "1", "created": 1793619658, "done": false},
  "parts": [
    {"fields": {"kind": "title"}, "text": "Embed documents from NSQ into the vector index"},
    {"fields": {"kind": "description"}, "text": "..."},
    {"fields": {"kind": "comment"}, "text": "..."}
  ]
}

collection is the Qdrant collection, as realdb names it (<project>-<table>).
id is the document, unique within its collection.
fields go onto every point of the document; each part's fields go onto that part's points as well. A part may not set a field its document sets.
Each part's text is cut at line breaks into chunks of about 1200 characters, and each chunk is one point.
Every point also carries document (the id), hash (of all the document's chunks) and hash_fields (of the fields of the document and its parts), which no field may set.

A document whose chunks and fields hash the same as every point already held is not written at all. One whose chunks alone hash the same is not embedded: only the fields of its points are written. Any other is embedded and replaces every point the document had, chunks it no longer has included. A document with no text removes its points.

Guarantees

Every embedder.* topic is read through the channel embedder, all of them side by side, so a producer with a long backlog never holds another up. embedder lists nsqd's topics every minute, so a new producer's first document waits at most that long.
Each topic's messages are handled one at a time, in the order nsqd hands them out. Nothing is deduplicated.
A message that is not a document, or a document that is invalid, is dropped and logged at error.
Any other failure (the embedding service or Qdrant down, a collection missing) hands the message out again after the consumer backs off, as many times as it takes. A document stays on the queue until it is written.
Embedder asks nsqd for a one minute message timeout and touches the message while it works, so a long document is never handed out twice at once.

Collections

A collection embedder writes declares, in its project's realdb schema, the two vectors it fills and an index on document, which every write filters by:

[tables.tasks]
vectors.dense       = { size = 384, distance = "Cosine" }
sparse_vectors.bm25 = { modifier = "idf" }
document            = { type = "keyword" }

bm25 is computed by Qdrant from each point's text. Nothing in embedder names a collection: its token, which rdb_qdrant.toml declares, reaches every collection on the server, so a new producer only has to publish.

Running

embedder with no arguments runs the service until interrupted, serving on port 8080 a page of what it is doing (GET /), the state that page polls (GET /state?window=<seconds>) and GET /health.

Every message handled is a row of the ClickHouse table embedder.outcomes, which rdb_clickhouse.toml declares: when it finished, its topic and document, what became of it, the tokens and time its embedding cost as the embedding service reports them, and its error. The rows are written every 10 seconds in one insert, and once more on the way out.

The page reads every count and rate from those rows over the window picked in its bar, 1m, 5m, 15m or 1h (keys 1 to 4). For each topic: how many documents wait, the document in hand, what became of its documents and the tokens per second embedded. Above them the totals, with the share of embedding time spent waiting in the embedding service's queue, which says the service is the bottleneck, and under them the window's last failure. task dev runs it from source. It runs on Coolify as the image docker.site.quack-lab.dev/embedder, which co_embedder.toml declares and bare task builds, pushes and deploys.

Variable	Meaning
`EMBEDDER_QDRANT_KEY`	token of the Qdrant user `embedder`, which `realdb sql` prints
`EMBEDDER_CLICKHOUSE_PASSWORD`	password of `embedder_embedder`, which `rdb_clickhouse.toml` sets from the same variable
`EMBEDDER_LOGS_WO_PASSWORD`	password of the log writer `logs_embedder_wo`; unset means console logging only
`EMBEDDER_SNIFFER_STREAM`	the log stream, `embedder` by default: the ClickHouse table `logs.embedder`

Each document written is one info line naming its collection, id, point count and whether it was embedded.

Layout

main.go              signal handling, Execute, close the logger
cmd/root.go          the command: config, cylogger and its sinks, Serve
internal/serve.go    wires the queue to the index and the dashboard, shutdown
internal/record.go   writes the finished messages to the outcomes table every 10s
internal/dashboard.go  the page, its state and the healthcheck
internal/ui          the page, built by bun with quacklab-ui and embedded
internal/index       documents to points, the hash check, Embedder and Store
internal/tei         the Embedder on text-embeddings-inference
internal/qdrant      the Store on Qdrant
internal/queue       the NSQ consumers of every embedder topic, handing documents to the index
internal/stats       the topics read and the document each has in hand
internal/outcomes    the outcomes table in ClickHouse, and its sums over a window
internal/rest        requests over HTTP for tei, qdrant, nsqd and ClickHouse