Deployment

Start with one,
grow to a mesh

The same binary serves both shapes. There is no separate cluster build, and no coordinator process to run beside it.

  • One node, the whole engine Storage, search and MCP in one process: at the edge, on a laptop, and in most production.
  • A mesh with no leader Consistent hashing places the data and a Kademlia DHT finds peers, so a node joins without an election.
  • Searches fan out, writes go home A search runs on every node in parallel and merges on the way back. A write goes to the node that owns its key.
GET /_cluster/health · output abridged
{
  "status": "green",
  "total_nodes": 1,
  "connected_nodes": 1,
  "cluster_total_shards": 4,
  "active_shards": 4,
  "queue_depth": 0,
  "predicted_wait_ms": 0, …
}

Health needs no credential, even with authentication on: a probe never carries a secret.

Containers

A distroless image,
amd64 and arm64

The image carries the server, the CLI and the MCP endpoint, like the binary. It runs as a non-root user with no shell inside to attack.

  • Two ports 9480 serves the HTTP API and MCP. 9580 is how nodes find each other, and a single node never publishes it.
  • Config and data, nothing else A TOML file at /etc/cameodb/cameodb.toml, read-only, and a data directory at /data/cameodb.
  • Digests, never keys CAMEODB_API_KEY_HASH carries a SHA-256 digest, so the environment never holds a key.
docker
# the image runs as uid 65532, so the data directory is its own
mkdir -p $(pwd)/data/cameodb
chown -R 65532:65532 $(pwd)/data/cameodb
docker run -d --name cameodb-server \
  -p 9480:9480 -p 9580:9580 \
  -v $(pwd)/data/cameodb:/data/cameodb \
  cameodb/cameodb:0.3.10

The same image is the CLI: --network host and client --interactive open a REPL against a running node. A three-node cluster is one compose file: each node takes CAMEODB_SEED_NODES and finds the rest itself.

Services

A service
on every platform

The deb and rpm packages run CameoDB under systemd, and the msi installs it as a Windows service. Config and data carry over through every upgrade.

  • Checked before it starts The systemd unit runs check-config first, so a config that fails it never opens the port.
  • Restarted after a failure Restart=on-failure under systemd; on Windows, ten seconds after the failure.
  • Reloaded, not restarted systemctl reload cameodb picks up new keys and renewed certificates in place.
Linux · deb, rpm
cameodb.serviceruns as the cameodb user
/etc/cameodb/cameodb.tomlconfig
/var/lib/cameodbdata
journalctl -u cameodblogs, or a daily file via [logging]
Windows · msi
NT SERVICE\CameoDBa virtual account, no password to manage
C:\ProgramData\CameoDBconfig, data and keys
…\logs\cameodb.YYYY-MM-DD.loga daily log, kept 14 days
Packages and install commands →
Security posture

The profile is a claim
the config must meet

A profile states who can reach the node. A config that contradicts its own claim refuses to start, rather than coming up quietly weaker than you meant.

  • local, internal, external This machine, a trusted network, or anywhere. Each one raises the bar on TLS, CORS, admin routes and keys.
  • One line per rule cameodb check-config answers pass, warn or fail, and exits non-zero on a failure: a pre-flight for any deploy.
  • Every failure says the fix Each line names the setting to change, so the next run passes.
cameodb check-config · output abridged
Security profile: external
  [PASS] bind         0.0.0.0 (reachable off-box)
  [FAIL] tls          profile 'external' requires TLS. Set
                       [network.http.tls] enabled = true …
  [FAIL] cors         profile 'external' does not allow
                       cors_allowed_origins = ["*"] …
  [FAIL] admin_api    … Set [network.http] admin_enabled = false
  [WARN] audit_trail  no audit trail: this node cannot say
                       who read which index …
  [FAIL] auth         profile 'external' requires authentication …

Result: FAILED
Keys and limits

Who, what
and how often

A key says who is calling, its index list says what they may reach, and the limits bound how much of the node one caller can spend.

  • Roles, pinned to indexes reader, writer or admin, scoped by allowed_indexes and refused on anything else.
  • Minted by your portal A provisioner key creates and revokes users’ keys through the node, and reaches no data itself.
  • Metered, quoted and audited Tool calls and written documents per minute, tenant quotas on indexes and bytes, and an audit trail per key.
terminal
$ cameodb keygen --role reader --label support-agent --allowed-indexes books
cameo_v1_…

The key above was printed to stdout and is not stored anywhere. Copy it now.

Add to cameodb.toml:

  [[security.api_keys]]
  key_hash = "sha256:b4d633af4d4c…"
  role = "reader"
  label = "support-agent"
  allowed_indexes = ["books"]

The node keeps only the digest: a key lost is replaced, never recovered from a config file.

Under load

Reads keep flowing
during bulk writes

Storage work never runs on the threads that answer requests, and a request the node cannot finish in time is turned away before it starts.

  • A firewall for blocking work Redb and Tantivy run on their own thread pools, so heavy ingest never stalls the async HTTP runtime.
  • A writer per shard, on its own core Parallel ingest never contends for one writer, and the core budget follows a container’s CPU quota.
  • Refused at the door, never left waiting Searches and writes each get three quarters of the slots, so a flood of one leaves room for the other.
a search flood against a node capped at two requests
HTTP/1.1 503 Service Unavailable
content-type: text/plain; charset=utf-8
retry-after: 1

Too many concurrent search requests

The refusal comes before the request is parsed or queued, from the same backlog estimate /_cluster/health reports as predicted_wait_ms. Health and /_admin/* skip the gate, so a node shedding load stays easy to diagnose.

Stop and restart

Stops in order,
starts in milliseconds

A clean stop leaves nothing to replay. A crash leaves only what was in flight, so recovery time tracks that, not the size of the data.

  • Shutdown in five phases Sessions close, connections drain, every shard commits and flushes, then the pools and the coordinator stop.
  • No repair after a kill Each commit records Redb’s allocator state, so a shard reopens in milliseconds at any size.
  • Commits on a clock A write is searchable within about two seconds by default, and never waits for a commit to land.
systemctl stop · log abridged
Phase 1/5: Closing MCP sessions (timeout: 5s)...
Phase 2/5: Draining HTTP connections (timeout: 10s)...
Phase 3/5: Shutting down all shards (timeout: 60s)...
Phase 4/5: Shutting down read thread pool (timeout: 10s)...
Phase 5/5: Shutting down coordinator (timeout: 10s)...
Shutdown complete - process exiting cleanly elapsed_ms=44
kill -9 during a bulk load, then start again
Crash repair: quick (allocator state recorded per commit)
healthy after 437 ms
books2 total_hits: 15,600, searchable from the first request
Disk

Deletes give
the disk back

cameodb compact returns the space deletes freed, across every shard, and --dry-run shows what it will reclaim first.

  • Safe to interrupt Every step is a committed transaction. Ctrl+C finishes the files in progress, and the next run picks up the rest.
  • The index merges live /_admin/index/{index}/merge folds each shard’s segments into one while writes and searches carry on.
  • Retention you can review scripts/ops/retention-prune.sh clears documents past a cutoff, in steps you see before anything goes.
after deleting 5,158 of 16,559 books · output abridged
$ cameodb compact -c cameodb.toml --dry-run
Measuring 5 redb file(s), 5 at a time (dry run: nothing is changed)
Would take 68.5 MB -> about 30.4 MB: 38.1 MB reclaimable

$ cameodb compact -c cameodb.toml
Compacting 5 redb file(s), 5 at a time
  metadata.redb                     0.5 MB -> 0.5 MB  (63.7ms)
  shard-0b6bc5ba…/store.redb       17.0 MB -> 6.9 MB  (163.1ms)
  shard-19d886af…/store.redb       17.0 MB -> 7.9 MB  (221.3ms)
  shard-7deb00b4…/store.redb       17.0 MB -> 7.9 MB  (255.1ms)
  shard-cd8dde6f…/store.redb       17.0 MB -> 7.7 MB  (297.2ms)
Took 68.5 MB -> 31.0 MB: 37.5 MB reclaimed