@Untitled Contact @Untitled Contact: findings from the prod memory investigation on 2026-10-19, with a proposed fix. Julio, you know the SQLite pool and the /debug/sqlite tracker best, so your read on the cache change would help most.
What happened
The kernel OOM-killed seed-daemon on the prod gateway box (hm, 15.6 GB RAM, 8 cores) twice:
when (UTC) | daemon resident memory when killed |
|---|---|
2026-10-17 15:58 | 11.0 GB (anon-rss 11,019,092 kB) |
2026-10-18 05:14 | 9.6 GB (anon-rss 9,634,344 kB) |
Both were box-wide OOMs (constraint=CONSTRAINT_NONE, global_oom): the daemon plus everything else ran out of RAM, and the kernel picked the daemon as the biggest process. Docker restarted it each time (restart count is 3). Each restart means a few minutes of failed requests.
On 2026-10-19, about 27 hours after the last restart, the daemon was already back at 6.5-6.8 GB resident and growing, so another OOM was likely within a day or two.
What it is not
Not a Go leak. /debug/vars memstats: HeapAlloc 0.61 GB, Go Sys 1.72 GB. The memory regions the Go runtime labels itself ([anon: Go: heap]) hold about 0.9 GB resident.
Not a JavaScript leak. The three web_gateway processes looked bloated (0.9-1.6 GB, with very different heap sizes for identical workloads). A forced full GC through the Node inspector brought two of them to the same ~275 MB of live heap, and their RSS fell from ~1.4 GB to ~0.75 GB. That memory was garbage V8 hadn't collected yet (heap limit 2 GB), not retained data.
What the memory actually is
Breakdown of the daemon's 6.5 GB, from /proc/<pid>/smaps:
what | resident | how we know |
|---|---|---|
glibc malloc heaps (C code) | ~5.0 GB | 63 large anonymous regions; every one starts with a glibc |
Go runtime | ~0.9 GB | regions labelled |
binary, shared libraries, embedding model file | ~0.1 GB | file-backed mappings |
Reading the headers from /proc/<pid>/mem gives 97 glibc heaps in 27 malloc arenas, typically 5 heaps (about 320 MB reserved) per arena. The process has 196 threads. Cgo calls run on whichever OS thread Go picks, so C allocations spread across many arenas.
The smoking gun
backend/storage/sqlite.go, in both the write and read connection preludes:
PRAGMA cache_size = -262144;A negative cache_size is in KiB, so that is a 256 MB private page cache per connection. SQLite allocates it with malloc, which is exactly where the 5 GB lives.
The pool is max(runtime.NumCPU(), 12) = 12 read connections plus the writer, so the caches alone can reach about 3.3 GB.
The prod database is 14 GB (/data/db/db.sqlite), so every connection's cache eventually fills completely. Memory climbs steadily for the first day or two after each restart.
This memory is private and anonymous. Under pressure the kernel cannot shrink or reclaim it; it can only kill the process.
The same pages are usually also in the kernel's page cache (SQLite reads the file through it), so hot pages are held twice.
Contributing factors
PRAGMA temp_store = MEMORY. Sorts, temporary tables and GROUP BY spill into malloc'd RAM. Our heavy queries (directory listings with comment aggregation, citations, the RBSR maintenance hook) allocate large transient buffers.
glibc keeps peaks. Freed memory in a non-main arena is only returned to the OS from the top of a heap. With 27 arenas, each one keeps roughly its high-water mark. The process ratchets upward rather than shrinking after a spike. This is likely the gap between ~3.3 GB of steady-state caches and the 9.6-11 GB at OOM time. That split is inferred; it would take allocator instrumentation to measure exactly.
The extra web processes since 2026-10-16. We went from one web_gateway to three to absorb crawler load. That added about 2 GB on the same box, so the daemon now hits the ceiling sooner.
Why now. The 256 MB setting is older than May. What changed is that the database grew to 14 GB, so caches fill completely, and the web side took more RAM.
Proposed fix
Shrink the per-connection cache and read through mmap. Set cache_size to something like -32768 (32 MB) for readers and keep a larger cache for the single writer. Turn on PRAGMA mmap_size (for example 1-4 GB) so reads come from the kernel's page cache. That cache is shared by every connection and reclaimable under memory pressure, so it cannot cause an OOM. This is a small change in backend/storage/sqlite.go. Expected saving: about 3 GB of steady-state private memory. To do: measure query latency before and after on a copy of the prod database, since some queries may lean on the private cache today.
Cap glibc arenas. Set MALLOC_ARENA_MAX=2 in the daemon container's environment (SeedInfra compose). This bounds how far fragmentation can spread; it's a standard fix for cgo plus many threads. Validate on dev first and compare resident memory over a day.
Consider temp_store = FILE, or a PRAGMA soft_heap_limit, so a single heavy query cannot balloon the process. This needs a latency check too.
Mitigations until the fix ships
Smaller V8 heap limit for the web processes: NODE_OPTIONS=--max-old-space-size=768. Their live heap is ~275 MB, so this only makes V8 collect sooner. It should free roughly 1.5-2 GB across the three. Seed PR #1255 (about 32% cheaper SSR, pending) cuts allocation churn and offsets the extra GC work.
Per-container memory limits, so a runaway container is restarted on its own instead of the kernel choosing a victim box-wide.
Bigger instance, if we want headroom while the above lands. This is a Terraform change and needs a reboot.
Related finding: a 36-second write transaction
The daemon logged SlowQuery duration=36.0s at 2026-10-18 17:40 UTC. /debug/sqlite shows it was blob.(*Index).applyIndexedHook, which runs MaintainRBSRIndex (backend/hmnet/syncing/rbsr_index.go). It spent 0.012 ms waiting for the writer slot and held it for 36.12 s, so every other write on the daemon was blocked for that long. It is also a pattern, not a one-off: 491 runs in the tracker window, p90 0.6 s, p99 1.76 s.
The capture keeps only the first 250 statements, which took 127 ms in total (the materialized-scope scan, then key lookups). The rest of the 36 s happened past the capture limit, so this needs a profile of the hook. A guess to verify: its cost scales with blobs per batch × materialized scopes.
How this was measured
All of this is read-only on the prod host (ssh hm), and all of it is easy to repeat:
journalctl -k | grep -i oom-kill for the kills and their RSS
/proc/<pid>/status and /proc/<pid>/smaps for resident memory by mapping
the first 32 bytes of each big anonymous mapping, read from /proc/<pid>/mem, to identify glibc heap_info headers
nsenter into the daemon's network namespace and curl 127.0.0.1:56000/debug/vars and /debug/sqlite
for the web processes, SIGUSR1 to open the Node inspector, then process.memoryUsage() and HeapProfiler.collectGarbage over CDP (inspector closed afterwards)
Do you like what you are reading? Subscribe to receive updates.
Unsubscribe anytime