The dev-only HTTPS test proxy moves to backloop.dev 5.2.1, which keeps the proxy alive when a reply lands after the response was already answered — the failure mode that can take the proxy down mid-suite.
5.2.0 also began caching the public and private certificate material
separately, which moved the bundle from certs/backloop.dev-bundle.crt to
certs/private/backloop.dev-bundle.crt. scripts/backloop-ca-warm looked only
at the old path, so on the new version it exited non-zero, test and
test-sqlite silently degraded to “no extra CA”, and the lib-js integration
suite was back to failing its first request with
DEPTH_ZERO_SELF_SIGNED_CERT. The warm step now resolves the bundle from
either location, newest layout first, so it works across the bump in both
directions.
Note that the certificate itself does NOT come from the pinned tag: backloop
fetches it in a postinstall step, so any install gets the current one
regardless of the version pinned here.
The audit record of a successful call and the access usage counters (calls, lastUsed) are
written after the response is sent, by design. Tests that read them back right after the call
raced that write and failed on slower CI runners ([AINT], [AUDT], [SYER] [9C1A]). They now
re-read with a bound through a new pollUntil test helper (test-helpers, 1500 ms default so a
missing row reaches the assertion rather than mocha’s 2 s timeout); the same applies to [AUAB],
[ASTE] and [AUDI]. [VBV0] and [0BK7] check the audit row of a specific call instead of
whichever row came first, which let [VBV0] pass without checking anything. The [AUDT] filter
tests await the audit writes they start instead of leaving hundreds running into the next hook
(the [JBPZ] hook timeout), let earlier calls’ writes settle before resetting their spies (now
restored afterwards), and assert “not called” as a zero count. [ASFL] waits for the rows of all
three accesses before its leak checks. [WHBK] [WB07] captures the webhook’s state and arms the
retry’s answer before the retry timer can fire, and its retry interval now outlasts the stored
read. Verified by slowing both writes by 300 ms: the previous tests fail, these pass.
The oauth-client/<clientId> row in PlatformDB (replicated to every core) carried the app
account’s username (accountUsername). It now carries accountUserId; the
client_credentials grant resolves the canonical username on the account’s home core (the
only core that can serve it) and answers 500 when the account is not hosted there. Each
core’s master boot converts, once and before workers start, the rows of the accounts it
hosts (migrateClientAccountIds); rows of other cores’ accounts are left to their home
core, and the grant still reads the old shape meanwhile. Rolling back the account’s home
core after the upgrade makes client_credentials answer 500 for converted clients until it
is upgraded again or the client is re-created. Note: a CLI-created client_id IS the
account username, so the username remains in the row key; only an opaque client_id would
remove it. [OCU1]-[OCU7].
DynamicInstanceManager installs one set of process exit / SIGINT / SIGTERM hooks for all
instances instead of three listeners per instance (which piled up and kept every manager
alive), removes its temp config file once its child has exited, and stop() now calls back
when the kill is reported as an error event rather than thrown. Its 5 s force-kill fallback
now actually fires for a child that ignores SIGTERM (it tested proc.killed, which is true as
soon as SIGTERM is sent) and no longer holds the process open. The port probe resolves only
after its socket has closed (a race too narrow to test deterministically). [DIM4]-[DIM7].
DynamicInstanceManager reset its readiness flag only in stop(). After a spawned test
server died unexpectedly or was force-killed, the next ensureStarted() trusted the stale
flag and returned before the new server was listening, so the first request failed with
ECONNREFUSED (the [SYRO] sighting). The flag is now reset at every spawn, and an
unexpected exit after ready is logged at error level, which reaches the durable test log.
Exit, error and readiness messages only act for the current child (a stopped child’s late
message cannot mark its successor ready), and a child killed by a signal or exiting 0
before announcing readiness now fails the start instead of resolving it.
The port allocator now probes the address the server binds (127.0.0.1; a probe on
0.0.0.0 succeeds on macOS over a port another process holds on 127.0.0.1) and stays
within 10000-49151, below the ephemeral range, wrapping around instead of running into it.
[DIM1]-[DIM3].
POST /reg/access/:key converts an inline-token accept into a one-time shared secret
when the request asked for credentialHandoff: 'shared-secret': it creates the secret on
the user’s core authenticated AS the app token (in-process through a MethodContext on a
local core, over HTTPS to the platform-resolved core otherwise), stores only the key, and
never persists the token, which reaches the answering core in the POST handler’s memory
only. A create that cannot be done (shared secrets disabled or forbidden, the core
unreachable, an unparseable endpoint, or a delegation-derived token, which may not create
the hand-off secret) falls back to inline delivery with one warn line, never the token.
The state store drops the token whenever a handoff is written, so a hand-off state can
never carry both. The “which core hosts the user, decided by the platform and never by the
posted apiEndpoint” resolution is extracted from the consent check into
routes/reg/userCore.ts (resolveUserCore) and shared by both callers. [RA95]-[RA106].
The lib-js integration suite (components/externals) pointed
service.assets.definitions at the public assets site on GitHub Pages, so the
[ASTX] and [ACNX] lib-js tests failed with a connect timeout whenever that
site was slow or unreachable. The assets they read (index.json, the default and
sign-in-button CSS, the button HTML and messages) are now copied under
components/externals/test/fixtures/assets/ and served by the suite’s HTTPS proxy
at /test-assets/; config/libjs-test-config.yml points there.
[SYRO] leaves nock off[UPLD] (uploads middleware), [SDTP] (subdomainToPath), [SSOC] (SSO OIDC client
flow) and [SSOE] (SSO end-to-end) still handed supertest a bare express app.
supertest then listens on :: at an ephemeral port and connects to 127.0.0.1; on
macOS another process already bound to 127.0.0.1 on that port receives the request
(reproduced standalone: the specific bind wins). That is how [GY5H] and [SSOC11]
got a foreign 404, and a plausible source of the [SSOE] socket hang up, in full
runs next to another test matrix. All four now use listeningAgent(), like
coreRequest. [SYRO] also stops calling useNock(): it mocks nothing, and it
routed its requests to the real spawned server through nock’s mock socket.
Authorization-code, refresh-token and consumed-marker rows live in PlatformDB, which
is replicated to every core, and carried the username in clear whatever the
platform’s hashed-PII mode. They now carry the user id only: the core that serves
/oauth2/token (always the user’s home core; another core’s code row is refused)
resolves the canonical username from its local users index, so a renamed user’s
chain also follows the current name. A user deleted since the grant gets
invalid_grant (a consumed refresh token still leaves its reuse-detection marker),
and the expired-code orphan sweep skips them. At boot, the master removes the
username from this core’s live refresh rows and markers once, before workers serve
/oauth2/token; code rows (10 min) simply expire. Rolling back to an earlier
release after this upgrade makes refreshes of chains minted since then fail until
the user re-consents. [OAC-OK2], [OPI1], [OPI3]-[OPI10], [OE08], [OE28].
The delegation plugin gains an accesses.create hook that stamps
clientData.delegation = { kind: 'delegated-child', relId, delegate, viaAccessId }
when the authenticated access is a delegate token or itself such a child (wired
after the forge-prevention hook, reads the authenticated access only), and an
accesses.update hook that re-applies the stored marker whenever an update touches
clientData. The delete and update lifecycle guards now protect only the markers
the plugin owns (control, delegate token, invite capability, notify, and any
unknown kind); a delegated-child access follows the normal access rules. The
detach teardown deletes the relationship’s delegated-child accesses right after
the delegate token (store.findMarkerAccesses) and logs the count. A new
events.create hook (createDelegatedGrantGuardHook, fed the CMC gated trigger
types) and a check in the OAuth2 resolveUser refuse delegation-derived tokens
(isDelegationDerivedAccess) on the grant paths that do not stamp the marker.
[DCHD], [DLN01-04], [DUP01-04], [DAD05-07], [DUG04-05], [DDG01-03],
[DDCH1], [OE27].
build/test/ (compose file, start script, per-service config) described the v1
topology (InfluxDB 1.7.8, separate register and mail services, a config layout the
v2 image no longer reads) and could not run against the single v2 image; 2.0.0-rc.22
had already flagged it as stale. It is removed along with its .gitignore,
.licenser.yml and ESLint ignore entries. Running the release image locally is
covered by the install wizard (bin/init.js, which writes run-pryv.sh) and the
Dokku notes in INSTALL.md.
MethodContext drops token from originalQuery (what the audit trail records as
content.query) next to the auth it already dropped. Callers that ask an access to carry a
chosen token send it in the body, which is not recorded, so this is a guard against a caller
putting it in the query string rather than a fix for an observed leak. [MCQT].
A suite that could not bind its port failed with a bare EADDRINUSE, which reads
like a defect in the code under test; in practice another checkout on the same
machine, or a server started by hand on the canonical port, was holding it. The
new test-helpers/src/portHolder.ts reports the holding pid and command line and
names the setting to move the suite’s port to. Wired into the pubsub
broker-reconnect test, next to the existing check in the externals suite.
build/build, build/Dockerfile and build/scripts/build_name built an image
from the v1 base image and stamped .api-version themselves, a second mechanism
that could disagree with the tag a release is built from. Releases are built by
CI from the root Dockerfile, which stamps the version from the image tag, so
the legacy path is removed (it also still installed with --omit=optional,
which strips sharp’s native binaries). build/test/README.md now builds its test
image from the root Dockerfile.
just install keeps optional dependenciesjust install ran npm install --omit=optional, which removes sharp’s native
binaries (sharp ships them as optional @img/sharp-<platform> packages), so the
previews server failed to load (Could not load the "sharp" module) until
sharp was reinstalled by hand. A full test run then crashed that component at
load without printing a failing test. The recipe now installs optional
dependencies, like CI, the Dockerfile and the native deploy path already did.
/reg/access state (api-server/src/routes/reg/accessState.ts) moved from
PlatformDB access-state/<key> rows to the core-local cluster_kv store (namespace
access-request/), the master-held map MFA sessions already use. The PlatformDB
choice was made to share state across cluster.fork() workers (issue #67); the
replication to every core was a side effect, and it put the ACCEPTED token and the
username on every core’s disk. Nothing about a request is needed on another core:
the poll URL is the entry core’s own. New markDelivered() shortens the life of a
terminal state to access:terminalRetentionMs after the first poll that reads it.
The cross-worker regression test ([XS12]) now runs the cluster_kv master handler
in the test process for the forked children (clusterFixture option kvMaster), so
the children no longer open rqlite.
OAuth2 storage (oauth2/src/storage.ts): codes and refresh tokens are keyed by
the SHA-256 of their value (oauth-ac/, oauth-rt/<coreId>/, oauth-rt-used/<coreId>/),
and the code row no longer carries accessToken / apiEndpoint, only accessId and
the issuing coreId. The code grant reads the access back from local storage through
a new resolveAccess dependency and refuses a row issued by another core; a failed
exchange deletes the orphaned access through revokeAccessLocal (storage-direct),
and the master’s expired-code pass (extracted to oauth2/src/orphanSweep.ts) deletes
this core’s orphans locally. Migration: rekeyLegacyRefreshTokens runs in the master
after migrations and moves this core’s oauth-refresh[-used]/ rows to hashed keys;
consumeCode still reads a legacy oauth-code/<code> row once (to be removed in the
following release), and legacy orphans keep the HTTP self-revoke. INTERNALS.md no
longer describes a forwardIfCrossCore hop at /oauth2/token that never existed, and
names the client-secret hash correctly (bcrypt).
The type repository (business/src/types.ts) now tracks whether the published
dictionary ever loaded (everSucceeded / degraded, plus source, version,
embeddedVersion, lastSuccessAt, lastAttemptAt, lastError), exposed via
getEventTypesLoadState() and the instance isDegraded() / getLoadState(). The
fetch is now bounded by an AbortController timeout so awaiting it cannot hang.
At boot, methods/events.ts awaits the first fetch (a healthy core validates
against the published set before it serves, closing the previous startup window),
and on failure schedules a background retry with exponential backoff (5s → 5min,
unref’d) that self-heals the core once the endpoint is reachable, logging recovery.
The previous fire-and-forget .catch(warn) is gone.
The HFS worker had the same dictionary fetch as a fire-and-forget with no
.catch; with no global unhandledRejection handler, an unreachable endpoint at
boot could crash-loop that worker. It now catches and logs, running on the embedded
fallback (hfs is already fail-closed for unknown types, so it refuses rather than
under-validates).
An accept or refuse arriving through a capability is written with the capability
token, so its createdBy names the capability access: handleIncomingAccept,
the new handleIncomingRefuse and setRequestEventIdOnAccess now read that access
by id (mall.accesses.getOne, a primary-key read on both engines) and fall back to
the full access scan only when the hint is not a capability access (a hint naming
another invite’s capability resolves nothing). Invite state is written by
one helper (cmc/src/inviteState.ts) with a transition table, so a late write
cannot overturn a final state. recordAccepter / clearAccepter are removed; the
three revoke paths mark the invite instead.
nock 14 switches itself on when it is loaded and then sends every HTTP request of the
process through a mock socket. Mocha loads every spec file before running any, and a
suite filtered out by --grep never runs its own hooks, so a filtered run (the usual
way to debug one test) ran socket-level tests against the mock instead of Node. Suites
that mock HTTP now call useNock() (test-helpers/src/nockScope.ts). Root hooks switch
nock off at startup and around every test outside such a suite. If a test leaves it on,
the hooks switch it off and the run fails at the end, naming that test. Test-only
change.
The event-types catalogue is checked as a whole when it loads, which does not look
inside each type, so a type whose own schema was malformed loaded silently and every
event of that type was then refused with no visible cause (four such types shipped in
the published catalogue before it was repaired). Each type schema of the bundled
default list (once per process) and of the list after every download is now checked
against the JSON Schema meta-schema, a few milliseconds for the whole list, and each
malformed one is logged as a warning naming the type and the problem; a schema that
still fails when compiled (an invalid regex, an unresolvable $ref) is named once, on
first use. Behaviour is otherwise unchanged. The validator facade in utils gains
schemaShapeError(schema), which never throws.
A downloaded catalogue is now applied entry by entry (each type, extras, classes and
sets entry replaces the current one whole; entries not in the download are kept)
instead of deep-merged, the same rule scripts/update-event-types uses for the bundled
list. The HFS server now logs a failed catalogue download instead of leaving an
unhandled rejection.
On raw deploys the API process forwards HF series traffic to the co-located HFS
worker by piping the client request into the worker request and the worker answer
back into the client response, and .pipe() forwards destroy in neither direction.
A client aborting mid-upload left the worker request half-open until the worker’s
request timeout (300 s). A client leaving before the worker answered, or stopping
mid-answer, left the worker socket stalled with the unread answer, for good when the
answer (a series query) exceeded the socket buffers. Each hop now propagates the close
of the stream it forwards: a client request that closes before its body completed
destroys the worker request, the response hop runs under pipeline(), and a client
already gone when the worker answers gets nothing piped. A teardown the proxy caused
itself is logged at debug, not as an upstream failure.
A worker answering before the client’s body was complete (an access refused on a large batch) also stalled the rest of the client’s upload until a request timeout; the proxy now discards the unread body, as Node’s own server does, and releases the worker request. nginx-fronted deploys never ran this code.
components/business/src/types/event-types.default.json is what a core validates
event content against until its startup download of the catalogue lands, and for
as long as it runs when that download fails (the failure only logs a warning). It
was last refreshed in October 2023: 23 published types were missing (among them
the consent/*-cmc, concentration/* and encrypted/aes-256-gcm types) and 5
descriptions were behind. A core that could not reach the catalogue at boot served
on that list, and every core did so during its startup window.
The seed is now the published catalogue applied onto the previous seed entry by
entry: every published type (and extras, classes and sets entry) replaces the
seed’s, and entries no longer published are kept. A running core merges its
download into the seed additively, so it still accepts types removed upstream;
keeping them makes a core that cannot reach the catalogue validate exactly like one
that can. That keeps the three legacy density/g-dl, density/mmol-l and
density/mg-dl types, which the catalogue renamed to concentration/*. Dropping
them would refuse new series:density/* events and is left as a separate decision.
just update-event-types now re-vendors both the test fixture and the runtime seed
(it used to overwrite the seed with the download) and re-checks them;
just update-event-types-fixture is kept as an alias. The catalogue guard in
just lint and CI fails when a published entry is missing from the seed or differs
there, and it checks the seed against the test fixture when the published catalogue
cannot be fetched, so that part no longer depends on the network.
When a response failed after its headers were sent (for example an attachment
whose file read failed mid-transfer), the error handler still tried to write an
error status. That threw inside the async handler, an unhandled rejection that
takes a worker down under Node’s default, and the client waited forever for the
bytes its Content-Length announced. The handler now audits and logs the error as
before, then cuts the connection, so the client sees a broken transfer at once.
A failure while writing the ERROR audit record had the same effect on the error path of every request: the handler rejected before answering, so the request hung and the rejection was unhandled. The audit failure is now logged and the error answered.
On the download path, a failure while writing the success audit record after the
file was served was also left to reject unhandled; it is now logged. A file that
cannot be opened (a missing file) or whose read fails before the first byte answers
with a JSON error status and an error audit record, and no longer carries the
attachment’s Content-Type, Content-Disposition and Digest headers, so a browser
does not save the error as the file.
The rqlite PlatformDB conformance suite and the ACME integration test defaulted to
localhost:4001 and ignored config/test-config.yml. A checkout running rqlite on
another port therefore either skipped 89 conformance tests while the run still
reported green, or ran them (writes and deletes included) against whichever rqlite
answered on 4001, possibly another checkout’s. Both now resolve the URL from
RQLITE_URL, then the test config (including its storages__engines__rqlite__url
env mirror), then the default. Both fail, naming the URL, when that rqlite is
unreachable, instead of skipping. The helper that boots a child core for
multi-core tests falls back to the configured rqlite too.
An attachment download pipes the file read stream into the HTTP response, and
.pipe() does not forward destroy upstream. When the client went away
mid-transfer (or before the file was opened), the read stream stayed paused with
its file descriptor open for the life of the process, so enough aborted
downloads would exhaust the process file-descriptor limit. The response’s close
now releases the source, and a client already gone when the file is opened gets
nothing piped. .pipe() is kept on purpose rather than pipeline(): a source
error before the first byte must still leave the response usable, so the client
gets a proper error status and the error is audited. An aborted download writes
no audit record, as before.
test/event-types-flat.json is a hand-copied snapshot of the published
event-type catalogue. test/service-info.json serves it to the test platform and
the business unit tests read it directly, so it is what every content-validating
suite validates against. Nothing tied the copy to its source: it was refreshed
only when somebody remembered, and it had fallen two months behind: two published
types missing (encrypted/aes-256-gcm, encrypted/ecies-aes-256-gcm) and one
definition changed.
The drift was silent in both directions. A type added upstream was exercised by
no test until the copy was refreshed by hand, and a suite could pass against a
catalogue no deployed client is served. The version field does not detect it,
because the catalogue is republished under the same version.
The fixture is now re-vendored to the current catalogue, and just lint runs a
guard (scripts/event-types-fixture-guard, also a CI step) that compares it with
the published file and fails with the re-vendor command when they differ. The
comparison is on canonicalised JSON, so a reformat upstream is not reported as
drift, and it names the types added, removed or changed. The guard SKIPS when the
catalogue cannot be reached (raised as a workflow warning in CI): an upstream
outage must not redden every build, while an unnoticed two-month drift is worth
failing on. A 404 fails, since it means the catalogue moved.
just update-event-types-fixture refreshes the copy and re-checks it.
bin/backup materialised every collection before writing it, and walked the
events array a second time to find attachments. Peak memory was therefore the
size of the largest collection, which is the wrong shape for exactly the
accounts a backup matters most for.
Events and audit now flow through a lazy pipeline: the export source, the
snapshot/incremental timestamp filter and sanitize run one item at a time on the
way to the writer, and attachment references are collected during that same
single pass instead of a second full iteration. Both stores gained a streaming
producer next to the array one (events.exportAllStreamed, and
exportAllEventsStreamed on the audit interface) on PostgreSQL through a
server-side cursor and on SQLite through a statement iterator.
The streaming producer is optional and feature-detected, so this is additive: the audit interface’s required-method set is unchanged, an engine that does not implement it keeps the array path, and the import/restore contracts stay array-based. The backup file format does not change.
On PostgreSQL the audit export runs on the streamed-read pool
(auditReadPoolSize), not the audit write pool, because exporting one account’s
audit set holds a cursor for the whole collection. bin/backup is a separate
process today, so nothing can be starved by it; keeping it on the read pool
means an in-process backup trigger added later cannot regress that.
UserAuditDatabasePG.getEventsStreamed and getEventDeletionsStreamed looked
like streams and were not: on the first read() they ran one SELECT, held the
whole result set in memory and handed rows out one at a time. Memory was the
size of the matching audit set, not of a batch. They now read through
DatabasePG.queryIterable, a server-side cursor that yields fixed-size batches,
so an audit query over a large trail costs a batch rather than the trail.
Making the read a real stream is only safe once “the response went away” reaches
whatever holds the resource. A cursor-backed stream owns a checked-out pool
client for as long as it lives, and .pipe() does not propagate destroy
upstream: the client going away destroyed the response and the outermost
Transform while every boundary below swallowed it, leaving the source suspended
and its client checked out forever. A handful of aborted requests was enough to
starve the pool, which is why an earlier attempt at this had to be reverted.
Every wrap site on the response path therefore goes through one helper,
utils/streams.ts pipeThrough, which builds the chain with
stream.pipeline() instead of .pipe() so destroy is forwarded end to end.
Result.writeStreams destroys the chain when the response closes early. Keeping
the call in a single helper is what makes “no .pipe() on the response path” a
checkable property rather than a convention.
Proven at the pool, not at the stream: [AUAB] aborts an audit query mid-flight
and asserts the pool’s checked-out count returns to its baseline, and
[PGQI]/[RSAB] cover the cursor’s own release paths and the abort hook.
pg-cursor is a new dependency of the PostgreSQL engine.
Streamed audit reads run on their own connection pool, separate from the one
audit writes use. New PostgreSQL engine setting
storages.engines.postgresql.auditReadPoolSize (default 5);
auditPoolSize keeps its name, its default and its meaning, and now serves
writes and short reads only. This is not tuning, it is the condition that makes
the change above safe to run: a streamed read holds its connection for as long
as the client takes to drain the response, there is no server-side response
timeout, and every access can read its own audit trail by default. On a single
pool, enough slow or deliberately stalled readers would queue the audit write of
every request behind them, and a queued write is dropped once it times out, so a
handful of readers could silence a core’s audit trail. With two pools the worst
a reader can do is deny audit READS to other readers for as long as it holds
them, which costs visibility rather than the record itself. Operators who raised
auditPoolSize for read throughput should raise auditReadPoolSize instead.
[AUAB4] holds the read pool at its ceiling and asserts an audit write still
lands.
Holding a client for the length of a response rather than a few milliseconds
also made two connection-loss paths matter that did not before. A checked-out
client has no 'error' listener of its own, because the pool removes its idle
one while the client is out, so a backend that went away mid-read (restart,
failover, an administrator terminating the session) raised an unhandled
'error' event and ended the process; the cursor holder now listens for the
length of the hold. And cursor.close() waits for a readyForQuery that a dead
backend never sends, so closing the portal on that path parked the release
forever and cost the pool slot permanently; the portal is now closed only on a
healthy connection, which is the only case where there is one to close.
[PGQI7] covers both.
Two smaller release gaps close with them. A method that failed after registering
a stream but before the response was written left that stream flowing and
holding its resource, as did the elements queued behind one that failed while
being collected; both are now released. And on SQLite, the wrapper around a
statement iterator gained the return() that Readable.from needs in order to
close it, so destroying a streamed read now reaches the iterator instead of
stopping one hop short. Writes were not affected by that leak, because user
databases run in better-sqlite3’s unsafe mode, which disables its open-iterator
guard; closing the database is checked whatever the mode, so a leaked iterator
made that user’s handle unclosable, which is what account deletion and the
handle cache’s eviction both need, and the un-reset statement held back WAL
checkpointing for as long as it lived.
handleIncomingRevoke now deletes the access named by the arrival’s createdBy,
so that access is gone by the time the handler returns. The loop-avoidance test
resolves createdBy to a counterparty access, which then no longer resolves: a
re-dispatch of the same inbox event (retry loop, operator re-processing) would
fall through to the outbound handleRevoke with the peer’s foreign
content.accessId, fail cmc-revoke-counterparty-access-not-found and rewrite
the arrival’s status to failed – which an app reads as “the withdrawal did
not work”. A consent/revoke-cmc sitting on :_cmc:inbox therefore always takes
the incoming path now, whether or not createdBy still resolves. Sound because
inboxWriteHook refuses any write to that stream from an access that is not
counterparty-marked, so an inbox arrival is peer-delivered by construction.
Dispatch also keeps its in-memory copy of the event content in step with the
status: 'delivered' stamp it writes, so a handler that rewrites content
afterwards carries the status forward instead of dropping it.
New components/delegation/ plugin owning the :_delegation:* namespace:
forge-prevention on clientData.delegation (create + update), lifecycle protection
(delegation-marker accesses and :_delegation:* events are undeletable/unupdatable
via the generic APIs by any token, including personal), reserved-namespace write
protection, and internal-subtree read guards. The delegate token is a session-backed
personal access minted like the login flow, discriminated by a forge-protected
clientData.delegation marker whose ABSENCE is exactly what the genuine-login detach
gate checks. Post-invite handshake delivery is modeled as marker-authenticated
controlled-side method calls (not stream writes) so the namespace write-guard stays
blanket; activation is synchronous with idempotent re-accept, holding an
at-most-one-control-access-per-relationship invariant.
Hardening shipped alongside: the hidden plugin-internal namespaces
(:_delegation:_internal and :_cmc:_internal) are now excluded from wildcard *
event reads via the same seam that hides shared-secrets and emails — closing a leak
where a * read could return the delegation A-side mirror event (which carries a
control-channel bearer onto another account) and the equivalent pre-existing CMC
internal state. The CMC internal-read guard was also hardened for single-value and
logical-query forms. Fixed a latent production crash: relationship-id generation
required cuid, a devDependency pruned from production builds — switched to the
standard @paralleldrive/cuid2.
business/src/emails/tokens.ts now owns mintToken / hashToken /
hashEquals (moved out of operations.ts, no behaviour change) so the
registration challenge and the container verification share one
implementation.business/src/emails/challenge.ts: PlatformDB access-state rows keyed by the
sha256 of the lower-cased address (email-challenge/<hash> for the code and
then the proof, email-challenge-sent/, -daily/, -fails/ for the
throttles). Attempt accounting is exact under concurrency: the row is
consumed atomically and re-installed with setAccessStateIfAbsent, so
parallel guesses cannot share a counter read. Rows expire by TTL and are
removed by the master’s existing access-state sweep.business/src/emails/mailCapability.ts: describeMailCapability (static,
method-aware “is mail configured”, no SMTP probe by design) and
describeVerificationMail ({ enabled, reason, explicit }), the single
predicate behind the boot validator, isVerifyMailEnabled and
service.info, so the three cannot disagree. explicit uses boiler’s
getScopeAndValue to tell an operator-set verifyEmail from the shipped
default (default-file scope).config/plugins/config-validation.js: the auth:emailVerificationPageURL
REQUIRED_WHEN now fires only for an explicit verifyEmail: true; new
collectWarnings logs non-fatal findings at every boot (default-on
verification with missing keys).business/src/emails/registrationPolicy.ts: config getters for the gate.errors.factory.forbidden(message, data?) and
tooManyAttempts(retryAfterSeconds?, { message?, data? }) gained optional
trailing parameters; existing callers unchanged.bin/master.js seeds components/mail/templates when templatesRootDir is
empty and warns when the gate is on but the challenge template is missing.requireEmailProof runs before forwardIfCrossCore
(non-consuming), the proof is consumed after insertOne, and the founding
container event is seeded with the email-code provenance.schema/accountMethods.ts: the emails[].verificationMethod response enum
now lists email-code. Result schemas are asserted by the tests
(account.test.js validates account.update against this one), so the new
provenance value had to land there as well as in PROVED_METHODS.cmc: the composed mall handed to the CMC modules is now fully typed, which
surfaced a live defect — streams.delete was declared and called with a
params object where the mall takes the stream id itself.[MC01]-[MC09] range was shared by three unrelated suites;
the mail CLI moved to [MCL0x] and method-context to [MCTX1-3], leaving
MC to the multi-core suite.[EMCH] (challenge), [EMCR] (registration gate over HTTP),
[EMCX] (cross-core forward ordering), [EMCG]/[MLCP]/[VMPL] (config),
[CV-GA]/[CKCF] (soft-landing validator), [EMLK] (link format),
[MSEED5]/[MFCD5]/[MFCD6]/[MCLI7] (bundled templates), [AS18] (platform
conformance), [SN09] (service-info), [SSOLI7] (email-code links).The streams and access caches were invalidated BEFORE their backing DB write
committed, in the engine stream mutators (insertOne/updateOne/delete), the
accesses.delete method, and the local user-index rename/alias operations. That
left a window where a concurrent read of pre-commit data could re-populate the
cache with a stale entry that no later invalidation removed (the epoch fences
close the invalidation-during-read half of the race but not this one). Each
invalidation now fires AFTER its write completes. For accesses.delete the cache
unset is also now built from the authoritative access rows and issued even when
the entry is not cached on the current worker, so a deleted token can no longer
keep validating on a sibling worker until eviction (it is dropped cluster-wide via
the existing broadcast). Behavioural fix only; no API surface change.
The streams cache had the same race as the access-logic cache: a stream tree is
loaded from storage on a cache miss and inserted with setStreams, and a
concurrent invalidation (a local stream mutation, or a cross-process
cache-invalidation broadcast) landing during the read could have its continuation
re-insert the pre-mutation tree, served until the next bust. The per-(user, store)
monotonic unset epoch now fences the insert the same way: the producer captures the
epoch before the storage read and the cache skips the insert if any invalidation
moved it meanwhile.
When an access is loaded from storage and inserted into the per-user access-logic cache, a concurrent invalidation (a local access update/delete, or a cross-process cache-invalidation broadcast) landing during the storage read could have its continuation re-insert the now-stale entry, which then served stale authorization state until the next eviction. A per-user monotonic “unset epoch” now fences the insert: the caller captures the epoch before the read and the cache skips the insert if any invalidation moved it meanwhile. The request’s own freshly-read access is unaffected; only the shared cache is guarded.
The cross-process cache-invalidation channel is a correctness requirement whenever API workers are forked (a cache bust in one process must propagate), so it has no configuration gate. Removed a dead always-true conditional and documented the always-on intent.
Runtime-dependency security bumps, all within the existing semver ranges: multer
2.2.0 → 2.3.0 (three high DoS advisories: crafted multipart field names, file-descriptor
leak on aborted uploads, oversized array index in field names), nodemailer 9.0.3 →
9.1.1 (high addressparser O(n²) DoS plus the moderate recipient-domain / punycode
validation bypasses), sharp 0.35.3 → 0.35.4 (high libheif advisories), morgan 1.8.x →
1.12.1 (moderate log-forging via unescaped Unicode line separators). npm audit
--omit=dev is now clean (0 findings) and the security-audit CI gate passes.
When an accepter holds several data-grants for the same peer and scope (a new grant is
minted on every accept, e.g. after a revoke-and-re-invite), the relationship selector
returned the first name-sorted match, so every inbound back-channel stamped the OLDEST
grant: newer grants never received a peer endpoint and their revoke could not notify the
peer (peerNotified: false, cmc-revoke-no-peer-endpoint), while outbound traffic routed
through a grant whose token had since been deleted (403). When more than one grant has the
exact scope, the selector now picks by created (newest first): an inbound back-channel
stamps the newest grant still awaiting one (never clobbering a completed relationship), and
outbound delivery goes through the newest grant that already knows the peer. Single-grant
relationships are unchanged.
The accesses.delete post-hook clears a withdrawn subject from an open-link
capability’s acceptedBy when a stamped relationship access is removed by a
plain accesses.delete. It was wired with the raw Mall, which exposes streams and
events but not accesses, so that local clear was a silent no-op: after a
requester removed a back-channel access directly, the subject stayed in
acceptedBy and re-consent through the same link was refused. The hook now
receives the composed CMC mall (an adapter over the accesses storage plus
token-auth cache invalidation), extracted into a shared helper so every CMC
wiring site uses it. A real-core integration test now covers the raw-delete path
the fake-mall unit tests could not.
The OAuth2 authorization accept drives a CMC consent handshake and polls for the
resulting data-grant. The handshake creates the data-grant BEFORE delivering the
accept to the peer and rolls it back if the peer refuses (an invalidated or consumed
link). The poll trusted the data-grant the instant it appeared, so under CPU
contention it could observe the transient grant during the create-to-rollback window
and mint a short-TTL OAuth access against a consent that was about to be refused: a
200 with a valid code where a 400 invalid_grant was due, leaving an orphan access
minted through an invalidated link. The poll now keys on the trigger’s terminal
status (the dispatch stamps completed / failed) and resolves the data-grant only
once the accept is completed, so the transient window is never observed. Extracted as
a pure awaitConsentOutcome helper with a deterministic, load-independent regression
test. A peer-refused accept still returns 400 invalid_grant carrying the peer’s
specific reason; on a transient (retryable) delivery failure the accept now returns a
server error until a background retry completes the consent, rather than optimistically
returning 200 off a data-grant whose delivery had not yet been confirmed.
The lib-js integration suite proxies lib-js over HTTPS on l.backloop.dev, with
the proxy’s certificate provided by backloop.dev. When no backloop secret is
configured (as in CI, and on a fresh clone), backloop serves its public
self-signed leaf, which Node’s built-in fetch rejects
(DEPTH_ZERO_SELF_SIGNED_CERT) so every lib-js suite failed at its first
request. The test and test-sqlite recipes now materialize backloop’s
certificate bundle (scripts/backloop-ca-warm) and trust it via
NODE_EXTRA_CA_CERTS before mocha starts, so the suite passes with or without a
secret. NODE_EXTRA_CA_CERTS only adds to the default trust store, so a
secret-based cert keeps working unchanged; the step degrades to no extra CA when
backloop cannot provision, leaving unrelated component runs unaffected.
An update key addresses at most ONE level inside a JSON field, and both engines
now enforce that through a single shared helper (storages/interfaces/_shared/updatePath.ts,
carrying the contract as its doc comment). Previously a key with two dots meant
different things per engine and said nothing about it: PostgreSQL split off the
first segment and merged the remainder as a LITERAL top-level key (data gained
an entry actually named a.b), while SQLite walked every segment and nested
properly. The same write therefore produced different documents, and a read
could not agree across engines. Such a key is now refused on both, through the
normal callback error and leaving the row untouched. The object form
({ data: { k: v } }) is unchanged and still merges one level with LITERAL
sub-keys, so an application key that legitimately contains a dot keeps working;
PostgreSQL no longer re-encodes it into a dotted string on the way through,
which would have made the new check reject it. PostgreSQL also gains one-level
$min/$max support, which it silently lacked where SQLite had it. To change
something nested deeper, read the entry, modify it, and set it back, or give it
its own entry. [PDOT] covers the semantics on every engine the suite runs.
Handed a bare express app, supertest binds a fresh ephemeral port for every
request and tears it down after. Across a full suite that was thousands of
bind/close cycles, and the resulting port churn was a genuine source of
cross-talk: a request could reach a port that had just been recycled and read a
response belonging to another listener on the machine, or a desynchronised one.
It presented as unrelated suites failing at random under load (spurious 404s,
socket Parse Error, hook timeouts) while every one of them passed in
isolation, which is why it survived so long as an apparently unfixable flake.
The global agent, the CMC/OAuth2 fetch shim and the per-suite agents now all run
against an already-listening server, via listeningAgent() which caches one
server per app instance so a suite that builds its own application still gets
its own. Full matrix went from 3 failing (PostgreSQL) and 5 failing (SQLite) to
0 on both.
startListen passed config.get('http:ip') straight to listen, so an unset or
nulled value fell through to Node default of every interface, silently publishing
the API to the network. It now falls back to 127.0.0.1, matching the guard the
HFS host already had.
A recovery code bypasses the second factor, but the codes were stored in the
clear on the user profile while the TOTP secret next to them was encrypted. Codes
are now stored as SHA-256 digests and the plaintext exists only in the response
that shows them once; a profile read back from storage cannot hand any code out.
A plain digest rather than a password KDF is deliberate: these are 122-bit random
values, so there is no guessable keyspace for a slow hash to defend and a KDF
would only add latency to every verification. Verification is constant-time over
all entries with no short-circuit, and accepts both shapes, so enrolments made
before this change keep working; those legacy cleartext codes disappear as soon
as the user re-enrols or recovers, both of which remove them. [MRC] covers it.
mfa.recover is deliberately NOT subject to the per-account limiter, in either
of its steps, and this is now pinned by test ([MA12H], [MA12I]) rather than
left as an accident of where the gate was placed. It is the last-resort path: the
recovery codes are 122-bit random values so a ceiling buys no security against
guessing them, while the password check is the same one auth.login performs
unthrottled, so limiting it here would remove no capability from an attacker and
would instead let anyone, with no credentials at all, lock a known user out of
their own recovery by submitting wrong passwords. The endpoint therefore never
reads or writes the throttle on failure and never returns too-many-attempts; a
SUCCESSFUL recovery still clears it, since the enrolment it guarded is removed.
Separately, the recovery-code comparison now runs in constant time over every
stored code (timingSafeEqual, no short-circuit) instead of Array.includes, so
neither the response time nor the position of a match is observable ([MA7E]).
[MA7D] pins that a wrong password and a wrong recovery code stay uniform for an
existing user. No new config and no new error id.
normalizeMfaConfig gained an attempts block (perSession/perAccount/
perAccountWindowSeconds/lockoutSeconds, defaults 5/20/900/900, applied in both the
new-model and legacy-mode branches; an absent or unusable field falls back to its
default rather than coercing to 0, which would silently disable the limit it
governs). The former MAX_MFA_ATTEMPTS constant is gone; the per-session ceiling
reads attempts.perSession. A new per-account counter lives on the user’s private
profile at data.mfaThrottle = { count, windowStartedAt, lockedUntil }, a SIBLING of
data.mfa rather than a field inside it: the profile store expands only one level of
dot-notation, and a deeper path is not portable across storage engines, so a
one-level sibling key is the shape that behaves identically on PostgreSQL and SQLite.
Keeping it outside the enrolment blob also means a routine data.mfa rewrite cannot
reset an accrued count as a side effect, so every clear is explicit: on a successful
verify/confirm, and on all three recovery paths (mfa.deactivate, mfa.recover, and
the admin system.deactivateMfa, whose $unset now covers both keys). The counter is
per-core, which is complete rather than a compromise: a user is pinned to one home
core, so that core’s profile sees all of their failed second-factor attempts and no
cross-core state is introduced. The verify/confirm paths gate on the lock BEFORE
verifying (no write while locked, no code-correctness leak) and mfa.challenge
honours the lock without accruing. A breach logs one warn (on the transition only,
never per guess) and returns the new too-many-attempts (429) error, which also
enters the anonymous telemetry error vocabulary and the existing per-user audit row.
[MA12A-F] cover the reported reproduction (fresh logins no longer buy a fresh
budget, and a correct code is refused while locked), auto-recovery, the
perAccount: 0 escape, per-session accrual, recovery-code unlock, and enrolment
integrity across throttle writes; [MNORM10-13] pin the config normalization.
storages.engines.rqlite.readyTimeoutMs) + a slow rqlited start is now visible in the logbin/master.js waited for rqlited’s /readyz with a hardcoded 30 s budget, on
both the managed path (rqliteProcess.start()) and the external: true path. A
node where rqlited legitimately needs longer (large platform dataset, slow disk)
failed boot with rqlited did not become ready within 30000ms, was restarted by
the supervisor and crash-looped until one attempt happened to land under the
limit; there was no config key to raise it. The budget is now
storages.engines.rqlite.readyTimeoutMs (default 30000, so existing deployments
are unchanged; it must be a positive number of milliseconds, otherwise master
refuses to start with a message naming the key). While waiting, master logs a
warning at 50% and 80% of the budget, and on success it logs the elapsed time
(rqlited HTTP API ready in 27.8s), so an operator can see a node running close
to the limit before a boot fails. The timeout error now names the probed URL and
the key to raise. The key is declared in the engine manifest, documented in
config/default-config.yml, offered as a commented knob by bin/init.js, and
shown in INSTALL.md. [RQREADY] unit tests drive the poll loop against a local
HTTP stub (success with elapsed time, 50%/80% progress warnings, timeout,
connection refused, external path) and pin the value resolution.
(open-pryv.io#127)
config/default-config.yml now ships services.mfa.active: true (TOTP default,
SMS off) so authenticator-app MFA works out of the box. To make the flip
upgrade-safe, normalizeMfaConfig (business/src/mfa/index.ts) precedence was
reordered: N0 explicit active:false wins, then a legacy mode shim (N2, now
ABOVE N1) so a mode: single|challenge-verify config keeps its SMS-only
behavior byte-identically instead of silently dropping to a TOTP-only model
where its SMS users would have no active method (a silent MFA bypass the flip
would otherwise have caused on upgrade), then the new active:true model (N1).
service.info().features.mfa = { methods: [...] } (api-server/src/methods/service.ts,
default-method-first, [] when off, absent on older cores) lets clients render
only active methods. Note: with MFA active, login.ts mfaCheckIfActive now does
one profile read per login that was previously skipped when MFA was off. Tests:
normalizer precedence [MNORM8/9], disabled-vs-default [MA1]/[MA15],
features.mfa [SN05-07].
The MFA subsystem grew a small MfaMethod interface + a per-method registry in
components/business/src/mfa/index.ts, alongside a normalizeMfaConfig that maps
both the new active/defaultMethod/methods shape and the legacy mode onto
one normalized form (rules N1/N2/N3; a one-time deprecation WARN on legacy
mode). The existing HTTP-provider SMS services are exposed through the interface
via a thin adapter; the HTTP Service base class is untouched. A new in-process
TotpService implements RFC 6238 over a dependency-free primitive (totp.ts:
Base32 codec + HOTP + constant-time verify with a drift window and a per-user
lastUsedStep replay guard), pinned by the RFC 4226/6238/4648 golden vectors.
TOTP secrets are encrypted at rest by reusing the shipped AES-256-GCM
AtRestEncryption envelope, under a key resolved from services.mfa.methods.totp.secretsKey
or derived from auth.adminAccessKey. The MFA profile and session store now carry
the method/totp state and a per-session attempt counter; the API layer resolves
the method per user, sets confirmedAt at confirm, persists lastUsedStep at verify
before releasing the token, and enforces a 5-attempt session limiter (all methods).
The SQLite events store recognised a duplicate id insert by an exact match on
the driver’s error message string. That is brittle: if the message shape ever
changes, the branch would silently fall through to unexpected-error (a 500)
instead of item-already-exists (409). It now matches on the stable SQLite
extended result-code (SQLITE_CONSTRAINT_UNIQUE) scoped to the id constraint by
substring, in both create and update. No API-facing behaviour change: a
duplicate event id still returns 409 item-already-exists, on the concurrent
path as on the sequential one. Added Pattern-C coverage ([EDUP1..3]: concurrent
HTTP smoke, engine-contract, mall pass-through) exercising both engines, since
the events chain has no existence pre-check and relies entirely on the store
constraint mapping.
An open-link capability records each accepter in
clientData.cmc.capability.acceptedBy to dedup re-clicks. Nothing cleared that
record when the relationship was later revoked, so a withdrawn subject stayed
listed and could not re-consent through the same link. The revoke paths now
clear the matching accepter entry (only that one — co-accepters are preserved,
and the capability state is never touched): the requester’s local revoke
(handleRevoke), a raw accesses.delete of the relationship (the delete
post-hook, which gains an optional mall dep), and a new handleIncomingRevoke
wired into the dispatch middleware for the peer-delivered case (local-only, no
outbound, with a legacy bridge that recovers the capability id from the revoke’s
offerEventId for relationships minted before the id was stamped). The
back-channel access now carries capabilityId so these paths can correlate the
relationship to its capability.
[OK] when it verified nothing--verify-integrity (and the standalone bin/integrity-check.js) derived its
verdict solely from error counts, so a run that checked nothing — integrity
inactive for a store, or the store unavailable — was reported identically to one
that verified everything clean ([OK] <user> — events=0 accesses=0, exit 0). The
integrity report now carries a per-store status (checked / inactive /
unavailable) and a top-level verified flag, and both CLIs distinguish
[OK] (verified clean) from [NOT VERIFIED] (nothing to compare against);
ok keeps its error-free meaning. New exit codes: 1 an integrity failure was
found (backup restore rolls it back), 2 restored/checked but could not be
verified, 0 fully verified clean. A restore whose verification is not fully
green now keeps its backup source (never runs --delete/move-on-success), and a
bin/backup.js restore that rolled a user back now exits non-zero instead of 0.
The integrity-preserving updateOne/delete on the accesses store wrote the
row hash-less in one statement and restored the hash in a second, as two
separate autocommitted round-trips. A concurrent full-store integrity scan on
another connection that landed between the statements observed the row without
its integrity hash and reported “access has no integrity property” — a
load-dependent false alarm (e.g. a back-channel data-grant update racing a
scan). Both statements now run inside a single transaction per engine
(PostgreSQL and SQLite), so no other connection can observe the intermediate
hash-less state, and a failure rolls back instead of leaving the row hash-less.
The accesses store also now requires its integrity reference at construction
(no silent inert fallback).
reconcileUserCoreMap now returns a conflicts list naming each local user or
alias whose routing row points at a DIFFERENT core (owned locally but claimed
elsewhere). Such names are never overwritten — they need manual operator
resolution — and were previously invisible: the tool silently declined to heal
them, so a dry-run counted them as would-be heals while apply healed nothing,
making the two disagree. The classification is now consistent across dry-run and
apply, and bin/reconcile-user-cores.js prints each conflicting name and the
core it is routed to.
On a multi-core platform every username check read only the core-local user
index, so a username hosted on a DIFFERENT core reported as available: each core
answered GET /reg/:username/check_username with reserved: false for names it
did not host, and the registry hostname round-robins across cores, so a client
got a wrong answer roughly half the time. Registration, the /system/users
reservation endpoint, and account.changeUsername shared the same blind spot,
so a name already hosted elsewhere could be registered or renamed into on a
second core, silently repointing the original user’s routing. See
https://github.com/pryv/open-pryv.io/issues/122.
A new platform-wide existence check consults the shared name-to-core map on
multi-core (single-core is unchanged by construction), and all the availability
call sites now use it. Registration reserves the username atomically for the
creating core (a redirecting core no longer writes the mapping, so a forwarded
registration is never pre-rejected), and changeUsername claims the new name
atomically as its cross-core collision gate. The name-to-core map is now kept
truthful: deleting a user (and each of its aliases) frees the routing row on
every core, and a failed registration releases its claim. A new operator tool,
bin/reconcile-user-cores.js, heals historical drift per core (removes
self-pointing rows with no local user, recreates missing rows for local users;
never touches rows owned by other cores). The engine layer gains an atomic
setUserCoreIfNotExists (claim-or-confirm) on both PostgreSQL and rqlite.
While here, two adjacent defects were fixed: registration now rolls back a
unique-field reservation if a later field in the same request conflicts (an
earlier value no longer stays reserved with no user behind it), and the
DELETE /system/users/:username?onlyReg=true admin route now deletes platform
fields through the mode-aware layer (the previous raw path read a unique
field’s value from an indexed row and bypassed hashing, so it left the unique
row behind and broke outright in hashed piiMode).
writeChunkedJsonlFiles sized chunks in compressed mode by probing
gzip(whole accumulated buffer) on every item once the raw size passed
maxChunkSize, and flushed only when that probe exceeded the limit. For highly
compressible data (audit logs, repetitive events) the gzip of the whole buffer
stayed under the limit, so the flush never fired: the buffer grew without bound
and the entire growing buffer was re-gzipped on every item, i.e. O(n^2)
synchronous compression that never completed. A full backup hung at ~100% CPU on
the first user with a large, compressible collection and never wrote a manifest
(issue #121). Chunks are now sized by raw (pre-gzip) bytes and compressed once at
flush, so total gzip work is linear and the in-memory buffer is bounded to one
chunk. maxChunkSize (and the --max-chunk-size CLI flag) now caps raw bytes
per chunk; gzip output stays at or below that, so on-disk files remain within the
target while compressible data yields smaller, more numerous chunks. See
https://github.com/pryv/open-pryv.io/issues/121.
Platform.init() guarded re-entry on the private #initialized field, but the
initialiser set the public initialized field instead, so the guard never
fired and init() re-ran its whole body (self-registration, core-URL cache
refresh, invitation-token seeding) on every getPlatform() call. That let
invitation-token seeding re-run mid-request whenever the platform config listed
tokens, seeding and then consuming a token into the platform database where it
persisted, which could make a later registration that reused the same token
fail with “Invalid invitation token”. init() now sets the guarded field so
its body runs exactly once.
Switch the Dockerfile base from node:24-bookworm to the digest-pinned
node:24-slim (Debian bookworm-slim). The build steps are unchanged (python3 +
build-essential + curl are apt-installed explicitly and then purged), so the
native modules (better-sqlite3, sharp) and the baked-in rqlite binary are
unaffected: a fresh image build passes the runtime smoke checks (rqlited
-version, and require of both better-sqlite3 and sharp). Slim carries far
fewer OS-package CVEs: a same-database Grype scan of the base images drops
Critical 60 -> 8 and High 248 -> 21. The built image’s residual findings are now
dominated by the application’s JS dependencies rather than the OS layer.
The build pipeline now detects, gates, and attests the software supply chain.
Added.
scripts/audit-prod-deps — a CI gate that fails the build on high or critical
advisories affecting the shipped (--omit=dev) npm tree. Accepted advisories
live in a documented, justified allowlist rather than being silenced.Changed. nodemailer bumped to 9, clearing the standing runtime advisories.
Known bound. The pinned node:24-bookworm base image still carries
OS-level CVEs that image scanners surface; a slimmer-base migration is tracked
separately. The claim here is detection, gating, and attestation, not a
CVE-free image.
revalidateConnections() is implemented on NamespaceContext, but the
30-second timer in socket-io/index.ts calls it on the Manager. The call was
undefined, so every tick threw manager.revalidateConnections is not a
function. The throw was caught and logged at warn, which is why it went
unnoticed on deployed cores while emitting two warnings a minute per worker.
A socket authenticates once at handshake and never re-authenticates, so two things can drop a revoked connection: the pubsub notification on an access change, which works, and this timer, which is the backstop for when that notification never arrives. Since a broker loss is exactly the case where it does not arrive (see the pubsub-reconnect fix in the same series), the belt was working and the braces had never been fastened.
Manager now implements revalidateConnections() as a fan-out over its open
namespace contexts, which is what the call site already assumed. [SNRS1-3]
pin the manager-level shape, the fan-out, and the empty case; without the fix
they reproduce the production TypeError exactly.
An incident responder who holds only a compromised accessId and a time window
can now produce the technical inputs for a breach notification (subjects
affected, records affected, categories of data by stream scope) without an O(N)
walk over every user or a fragile re-run of the historical query.
Added.
components/platform/src/accessIndex.ts) mapping a bare accessId to its
owning user plus minimal operational metadata (type, expires, created /
last-modified, deleted). PII posture follows the existing hashing regime: in
hashed mode the row carries the username HMAC token, never plaintext, and the
secret access token is never stored. Subject erasure removes or tombstones the
row. Populated by hooks at every access-creation site plus update / delete
cascade; a bin/backfill-access-index.js script backfills and repairs.GET /system/accesses/:accessId that normalises composite
<base>:<serial> ids and always returns revoked accesses (scoping a breach
after revocation is the primary case).content.recordCount (records the read delivered;
0 on empty reads, 1 for a single-event read, absent on writes and
pre-change rows) and content.scopedStreamIds + content.scopedStreamCount
(the streams the read was scoped to; a bare wildcard collapses to ['*'], the
id list caps at 100 while the count preserves the true total). A partial count
from an aborted drain is labeled recordCountIncomplete. The fields ride in
the audit content blob, so no schema migration.bin/breach-scope.js: a read-only, credential-free operator CLI run on the
subject’s home core. It resolves the owner, recovers the plaintext username
locally in hashed mode, pulls the access’s audit rows for the window, and
renders a caveat-labeled report as JSON and/or Markdown, plus a
breachScopeReport module that does the classification and rendering.Fixed. Socket.io API calls produced no audit row at all; they are now audited like HTTP calls (best-effort, never breaking the socket response). Batch calls drain the result before auditing so streamed reads finish counting.
The optional APM integration is rebuilt around a single choke point instead of an in-process vendor agent. Rationale, since it is the whole point: configuring an agent that auto-instruments everything makes the data-protection posture a property of that agent’s evolving defaults, provable only by enumerating what must not escape. Constructing telemetry ourselves makes it a property of two readable files.
Removed. The newrelic optional dependency; the provider boot shim
(bin/_observability-boot.js) and its require from all five entrypoints; the
provider adapter/config directory; the dead log-forwarder module; the
setTransactionName / recordCustomEvent / startBackgroundTransaction
façade surface and its call sites, including the named-transaction table in
API.ts and the background-transaction wrapper in CertRenewer. Because no
library is being patched, the “must be the first require in the process”
constraint is gone with it.
Added, all under components/business/src/observability/:
schema.ts — the allow-list: metric names, per-metric attribute keys, closed
enums, and the method-id and error-code vocabularies registered at startup.
Validation returns a drop reason rather than throwing.emitter.ts — the choke point. Validates, aggregates in memory (delta
counters and fixed-bound histograms), batches on a timer off the request
path, and drops-and-counts anything refused or unsendable under
telemetry.dropped. Bounded buffers; a failing backend costs a counter.sanitizeError.ts — class name plus stack frames, absolute paths rewritten
repository-relative, frames from outside the repository discarded, and the
message never forwarded.errorRegistry.ts — reuses ErrorIds as the code vocabulary, so emitted
codes are the ones the API already documents; decides which failures deserve
a stack (5xx, unexpected, unclassifiable) versus counting only.otlp.ts — OTLP/HTTP JSON payload builders, hand-rolled. No vendor SDK and
no OpenTelemetry SDK: an SDK is precisely the kind of dependency that
auto-instruments and widens the surface unnoticed.startup.ts — starts the emitter from the environment master resolved.Wiring. API.call is the single instrumentation site: method id, outcome
and duration are all known there, and it serves HTTP and socket calls alike.
Application.initiate starts the emitter after the routes are registered,
because the method registry is the emitter’s vocabulary. Platform’s
observability config swaps the vendor key for otlp-endpoint plus an encrypted
otlp-headers map, which is what makes the backend interchangeable.
Two defects that only a deployment surfaced, both fixed, and both worth recording because they are the failure grammar this layer was rebuilt to retire: telemetry that reports success while emitting nothing.
Server.registerApiMethods, which runs after
Application.initiate, where startup had been placed — so the emitter
came up knowing zero method ids and would have refused every datapoint as
unknown-method-id. The deployed log line read telemetry emitter active
(0 methods). Startup moved to the end of registerApiMethods, and
startFromEnv now REFUSES to attach on an empty vocabulary with a legible
reason rather than activating inert. [OB11] pins the refusal. The local
suite could not have caught this: its tests hand the vocabulary in
explicitly, and nothing asserted that the real wiring supplies one.telemetry.dropped said nothing about what was refused. The
drop path now names the offending value in the LOCAL log alongside the
reason. Local logs already carry identifiers, so this adds no exposure, and
it never becomes part of a payload.The reporting interval is now a real control. It was hard-coded while the
documentation already told operators they could widen it to reduce the
low-traffic correlation residual. observability set-interval <seconds>
stores it in PlatformDB, master resolves it with the rest of the posture,
workers receive it, and the emitter clamps it to 60-3600 so a
misconfiguration cannot make individual activity finely observable. Default
raised 60s to 300s.
Tests. [OBS1]-[OBSQ] (business unit suite) are the filter proof: they
ask the validator what it decided for accepted and refused inputs, including
a fuzz pass over identifier-shaped keys and values, with a legitimate datapoint
pinned in each block so the suite cannot pass by dropping everything. [OB01]-
[OB10] cover config resolution, encrypted round-trip, worker env propagation
and startup refusal; [OC01]-[OC08] cover the CLI.
On a fresh deploy the boiler logger’s only sink is a log file that may not exist yet; an invalid config then produced exit 1 with no output at all, forcing operators to wrap process.exit to find the caller. config-validation.js now factors the reporting into reportProblems(), which writes the header + every problem to stderr (tagged [config-validation]) in addition to the logger, before load() exits — stderr always reaches the operator (terminal, systemd journal, container stdout). Unit test [CVSE-01] asserts the mirror. Also: tools/performance/README.md now names rqlite instead of the removed “platform” SQLite engine.
attachmentsDirPath config knobstorages.engines.filesystem.attachmentsDirPath was declared required in the engine manifest and set in every config file, but read by no code path: event attachments co-locate with per-user data under the user local directory (storages.engines.sqlite.path) via UserLocalDirectory.getPathForUser(userId, 'attachments'). The knob was a leftover from before that refactor and actively misleading — paths-config and production-config pointed it at a separate directory that stayed empty. Removed from the manifest, default/production config, paths-config and bin/init.js (kept mirrored), documented the co-location in userLocalDirectory + the manifest previews description, and scrubbed the stale INSTALL.md examples (including the encrypted-volume one that recreated the empty-directory trap). No behaviour change — getPathForUser is untouched, so attachments land exactly where they already did; manifest.configuration.fields is declarative (not validated), so removing the field is inert.
Three internal gaps behind the operator-visible scrubbing change.
The agent never loaded the config file. It discovers configuration by
scanning NEW_RELIC_HOME for exactly newrelic.js, newrelic.cjs,
newrelic.mjs, or whatever NEW_RELIC_CONFIG_FILENAME names, and otherwise
falls back silently to environment variables and built-in defaults. The config
lived in newrelic.ts, which is invisible to that scan, and the provider boot
module only calls require('newrelic') without referencing it. Verified against
the pinned agent’s own loader: with the directory as it was, the agent resolved
attributes.exclude: [], record_sql: 'obfuscated' and
application_logging.forwarding.enabled: true. The configuration is now in
newrelic.cjs (.cjs because the package is ESM and the agent uses require),
with newrelic.ts re-exporting it so callers and tests read the same object.
[OBSC5] asks the agent’s loader what it resolved, so this class of gap fails a
test rather than passing silently: every other assertion in the suite inspected
our own exported object and was happy throughout.
The high-security opt-in never worked. The worker-env builder read
obs.newrelic.highSecurity, but the platform config resolver never returned
that field and no CLI command could set it, so the corresponding environment
variable was always false regardless of operator intent. The resolver now
reads a newrelic-high-security platform row (stored in the clear: it is not a
secret, and operators compare it across cores), a local YAML override wins over
it exactly as it does for enabled, and observability newrelic
set-high-security writes it. Covered by [OBHS]-[OBHS4], which assert the
row reaches the resolved config and the worker environment rather than just the
database.
Agent config constants are now test-pinned. The exclusion list, URL
obfuscation pattern, record_sql: 'off' and the log-forwarding default are
quoted verbatim in customer-facing data-flow documentation, so [OBSC]-[OBSC4]
assert each of them. A future edit that loosens the posture now fails a test
instead of silently contradicting published documentation. [OBSC4] loads the
config in a child process on purpose: the log-forwarding default is evaluated at
module load and these modules load through ESM interop, so clearing the module
cache in-process does not re-evaluate them.
Diagnostics. observability show now prints the high-security state and a
summary of what is and is not sent to the provider, and reports the
log-forwarding line conditionally since it cannot observe the service process’s
environment from another shell. When storage engine start-up fails, the command
now explains that engine start-up opens the user-data connection before it
reaches the platform database, instead of surfacing a raw driver authentication
error that sends operators looking in the wrong place.
The internal TCP pub/sub bus (used, among other things, to broadcast cache
invalidations across workers) had no reconnect path. A worker that lost its
connection to the broker — because the broker process crashed, was redeployed
or recycled, or because the worker lost the startup race and its first connect
failed — stayed disconnected for the rest of its lifetime, logging only a
warn. Since access-cache invalidations travel over this bus, a disconnected
worker never dropped its cached copy and could keep serving a revoked access
indefinitely; on a clustered deployment with requests spread across workers,
whether a deleted token still worked became a coin flip until a full restart.
Clients now reconnect with capped exponential backoff, re-run broker election
on each attempt (so a survivor takes over the freed port when the elected
broker dies), and re-send their subscriptions once reconnected. A failed
initial connect no longer strands the module in permanent local-only mode, and
a worker that has genuinely lost the bus now logs at error rather than warn.
No API or behavioural change on the healthy path. Covered by regression tests
that kill the broker mid-run and assert delivery resumes.
The method-chain runner ran each function inside a synchronous try/catch
without observing the promise it returned. A function that threw after an
await therefore rejected without anyone noticing: next was never called,
the callback never fired, and the HTTP request hung forever with no response
and a leaked socket — reproducible with a single request from any valid token
via GET /streams?parentId=:<unknown-store>:foo, where the store lookup
throws past an await. Such failures now propagate normally and return an
error response. A late rejection arriving after the chain has already
advanced is logged instead of advancing it a second time. No change on the
healthy path; failures that previously hung now behave like any other error.
When integrity is active for accesses, AccessesPG / AccessesSQLite now
assert, at write time in applyDefaults, that the access being persisted
carries a computed integrity value. If the integrity step is silently
skipped (for example because the storage layer was constructed without a
real integrity reference and fell back to the inert stub), the write now
fails immediately with a diagnostic error naming the access, the process id
and whether a real integrity reference was injected — instead of the gap
surfacing one operation later as an “access has no integrity property”
whole-store scan failure that is hard to trace back to its origin. No API
or behavioural change on the healthy path; this only makes an otherwise
silent inconsistency loud at its source. Covered by unit tests on both
engines.
The bin/init.js config wizard and bin/check-config.js now default
access.defaultAuthUrl / auth.passwordResetPageURL to the
app-web-user-account React app (routes /auth, /reset-password) instead
of the deprecated app-web-auth3 (/access/access.html, …). Fresh installs
point at the current reference auth+account web app out of the box. No runtime
API change; existing deployments are unaffected (their configured values are
untouched).
CMC orchestration previously treated consent/{accept,scope-update,revoke}-cmc
as authoritative user consent regardless of which access wrote them: an app
token with :_cmc:apps:<app>:*, contribute could write a consent/accept-cmc
event whose capabilityUrl pointed at a colluding requester offering
permissions: [{streamId: '*', level: 'manage'}], and the recipient’s plugin
would dutifully mint a shared data-grant access on the user with */manage
— no consent UI shown, no user-presence guarantee. The fix layers an
events.create gate for the access-mint/widen triggers, an access-permission
gate inside the revoke handler, and defense-in-depth chain checks in all
three orchestrators that mutate access state.
cmcAcceptAccessGateHook (components/cmc/src/cmcAcceptAccessGate.ts) is a
new events.create middleware. It rejects consent/accept-cmc and
consent/scope-update-cmc writes when context.access.isPersonal() returns
false, with cmc-accept-requires-personal-token (400 invalid-operation,
CMC id at error.data.id). Plugin-managed accesses
(clientData.cmc.kind === 'capability' or clientData.cmc.role === 'counterparty')
are explicitly exempted so the cross-platform handshake — bob’s capability
POST to alice’s :_cmc:_internal:responses:<capId> and counterparty
deliveries via the shared data-grant pair — continues to work. Reuses
AccessLogic.isPersonal(); no parallel “is personal” implementation.
Revoke is a contraction, not an escalation — the access being deleted bounds
the impact. consent/revoke-cmc is NOT in the events.create gate; instead
handleRevoke calls triggerAccess.canDeleteAccess(target) before each
mall.accesses.delete (data-grant + counterparty). This is the same
primitive accesses.delete uses, so the existing selfRevoke feature
permission carries over:
selfRevoke: allow) — no auth-page bounce needed;cmc-revoke-forbidden.Peer-delivered revokes never reach handleRevoke (dispatch’s
isPeerDeliveredEvent short-circuit on OUTBOUND_LOOPABLE_TYPES returns
'skipped' first), so the check doesn’t interfere with cross-platform
delivery.
The api-server’s accesses.{create,update,delete} routes enforce
access.can{Create,Update,Delete}Access(...) in applyPrerequisitesFor{...},
but the CMC plugin used to call mall.accesses.* directly — bypassing those
checks. All three handlers now run the same primitive before the
storage-layer call, no parallel implementation:
handleAccept → triggerAccess.canCreateAccess(dataGrantPayload) before mall.accesses.create. Failure: cmc-insufficient-permissions.handleSystemScopeUpdate → triggerAccess.canUpdateAccess(target) + triggerAccess.canCreateAccess({permissions: mergedPerms, type: 'shared'}) before mall.accesses.update. Failure: cmc-insufficient-permissions.handleRevoke → triggerAccess.canDeleteAccess(target) (see above) before each mall.accesses.delete. Failure: cmc-revoke-forbidden.The trigger-writer’s AccessLogic is plumbed through the dispatch’s
per-request deps (triggerAccess: context?.access) so the handlers can reach
it. handleIncomingAccept’s back-channel mint stays on direct
mall.accesses.create — its permissions are bounded by the original
request, which is chain-checked requester-side at publish time.
[CMCAUTH-*] 10 unit + 8 integration — the events.create gate (now covers accept + scope-update; revoke moved to a separate suite).[HR-AUTH-*] 5 unit — handleRevoke.canDeleteAccess matrix (personal pass, self-revoke pass, total-no reject, partial reject, skip-when-no-triggerAccess).[HS-AUTH-*] 4 unit — handleSystemScopeUpdate chain check (personal pass, canUpdate=false reject, canCreate=false reject, skip-when-no-triggerAccess).[CMCHS-*] 14 tests) + the full CMC integration surface stay green.User-facing impact: see CHANGELOG-v2.md “BREAKING — CMC trigger writes
that mint or widen accesses now require a personal token; revoke is
access-permission-gated” for the wire-level error contracts + the
@pryv/cmc.requestAccept / requestScopeUpdate hand-off helpers. No
requestRevoke helper is needed — apps holding the relationship access
self-revoke directly via the existing cmc.revokeAcceptance /
cmc.revokeRelationship calls.
New top-level component components/oauth2/ lays the groundwork for the OAuth 2.0 authorization-server surface. No public auth flow yet — /oauth2/authorize + /oauth2/token arrive in a follow-up — but the substrate is fully wired:
.well-known/oauth-authorization-server (RFC 8414) discovery doc is served from every core. The issuer + endpoints advertise the load-balancer-facing service URL, NOT per-core URLs, so the operator MUST keep the new oauth.* config block in sync across cores in a multi-core deployment. Discovery doc advertises response_types_supported: [code], code_challenge_methods_supported: [S256] (PKCE mandatory, no plain), authorization_response_iss_parameter_supported: true (RFC 9207), and the configured grantTypesSupported (starts with authorization_code; later milestones add refresh_token + client_credentials).registerScopeParser(namespace, parser)) shipping with the pryv: namespace registered (pryv:read / pryv:write / pryv:manage). Future namespaces (e.g. SMART on FHIR) layer on as plugins without migrating persisted scope data.error.id → RFC 6749 §5.2 error-enum map at the endpoint edge; unmapped error.ids fall through to invalid_request rather than leaking Pryv-specific strings to OAuth clients.oauth-client/<clientId> (allowed by the no-credentials-in-PlatformDB invariant — client_secrets stay on the app account’s home core). Cross-core /oauth2/authorize validates client metadata without round-tripping to the app’s home core. Exact-string redirect-URI matching with the loopback-port carve-out (RFC 8252 §7.3) — no regex, no prefix matching.oauth-code/<coreId>/<code> (600 s TTL, single-use) and oauth-refresh/<coreId>/<token> (sliding 30 d, hard-cap 90 d absolute). The coreId prefix locks codes + refresh tokens to their issuing core (the wrong-core path will server-side-forward /oauth2/token in a follow-up milestone — vanilla RFC 6749 clients never see the cross-core hop).bin/oauth-client.js operator CLI for app-account promotion in curated registration mode (the only supported value in this milestone; oauth.clientRegistration.mode: open is rejected at config validation). Subcommands: create <username> (promotion-only — refuses if the user account doesn’t already exist), show <clientId>, list, update <clientId>, revoke <clientId> --yes (the --yes gate is intentional operator-footgun protection).WWW-Authenticate: Bearer + DPoP challenge builders for the middleware layer’s 401 responses (RFC 6750 §3); the middleware wire-up arrives with the public flow.audit.ts) defines the event-type catalogue (oauth.consent.{shown,granted,refused}, oauth.code.{exchanged,reused}, oauth.token.issued.<grant_type>, oauth.token.refreshed, oauth.token.revoked) — emission lands when the public flow does.Covered by [OAUTH-SCOPE] [OAUTH-ERR] [OAUTH-CLIENT] [OAUTH-WK] (component unit tests) and [PLKV] [OAUTH-STORE] (PlatformDB conformance — both engines).
Adding a core could take a previously-healthy core’s control plane offline. A new core joined the Raft cluster as a voter immediately, so a two-core cluster ran at 2-of-2 quorum: if the new core (or any core) then became unreachable — crash, restart, redeploy, transient network — the survivor lost majority, stepped down, and platform reads/writes stalled. Under container orchestrators this was easy to trip, because a zero-downtime health check could start a new core, let it ack and join as a voter, then stop the container — stranding an unreachable voter. Two further snags made the documented join fail or surprise operators: the bootstrap ack pinned the cluster CA (so it failed against a core whose API is fronted by a public/ACME cert), and there was no guidance on the quorum math.
A joining core is now a non-voter by default: it replicates the platform DB and transparently forwards writes to the leader, but does not count toward quorum or vote in elections, so an unreachable joiner can never stall the cluster. The first/single core stays a voter (a lone non-voter cannot elect a leader). Concretely:
bin/master.js --bootstrap defaults to a non-voter join; pass --bootstrap-as-voter to join as a voter, which you should only do when building a 3-or-more-voter HA cluster. core.nonVoter (default false, set automatically for bootstrap joins) drives rqliteProcess.buildArgs(), which emits -raft-non-voter and omits -bootstrap-expect (the two together are fatal in rqlited). DNS-based peer discovery is unchanged.--bootstrap-ack-trust-system-ca verifies the ack against the system CA store instead of pinning the cluster CA, for the common case where the existing core’s API origin presents a public/ACME certificate. The one-shot join token remains the authenticator.bin/bootstrap.js promote-core <id> [--force] promotes a non-voter to a voter: it checks the target is a reachable, caught-up non-voter, refuses to create a fragile two-voter cluster without --force, then removes it on the leader so it can rejoin as a voter (rqlite has no in-place promotion).SINGLE-TO-MULTIPLE.md documents the quorum math (never run exactly two voters; use 1 voter + non-voters, or ≥3 voters), a topology→role preset table, the orchestrator deploy contract (disable zero-downtime health checks for cores that bind privileged ports directly, or front with nginx), and a peers.json single-node recovery note.Covered by [RQARGS], [APPLYBUNDLE] and [BOOTSTRAPCLI] unit tests. (open-pryv.io#99)
rqliteProcess.buildArgs() passed -raft-cluster-remove-shutdown unconditionally, so every node removed itself from the cluster membership whenever its process stopped. A process stop is not a decommission: crashes, upgrades, and container reschedules all trip it, so multi-core clusters shrank on every restart and were fragile under orchestrators that recycle containers routinely (the single-core case re-bootstrapped its lone node, which masked the bug in the common deployment). The flag is removed — a restarting node keeps its membership and rejoins via its existing node id and Raft log; permanently removing a node is now strictly a deliberate operator action, not a side effect of the process exiting. The per-worker parallel-test launcher carried the same flag and is cleaned up too. [RQARGS] argv tests updated, with a regression assertion that the flag is never emitted (single- or multi-core); the two-node mTLS cluster integration test confirms cluster formation + replication are unaffected. (open-pryv.io#98)
bin/backup.js outputBackupOrchestrator was calling auditStorage.forUser(userId).exportAllEvents() without awaiting forUser — which returns Promise<UserAuditDatabase> per the interface contract. The unawaited Promise has no .exportAllEvents method, so every per-user iteration threw userAudit.exportAllEvents is not a function. The surrounding try/catch downgraded the failure to a warn, and the backup completed EXIT=0 with zero audit data for every user. This had been live since 464ce266 (the events-export fix that first allowed the audit step to be reached). Restore from such backups silently dropped the audit trail — a problem for operators relying on it as part of their retained record.
The fix adds the missing await and replaces the orchestrator’s inline AuditStorageLike typedef with an import of the real AuditStorage interface, so a future drift between the orchestrator and the storage contract becomes a typecheck failure instead of a silent runtime warn. The catch is removed: when auditStorage is configured and the export throws, the backup now fails loudly (non-zero exit) — the silent warn was exactly how this lasted from 464ce266 to today. Operators who run without audit (engine declares no auditStorage) are unaffected — the if (this.auditStorage) guard already returned early there.
Audit data exported this way is intentionally raw engine rows (snake_case on PG, per the UserAuditDatabase.exportAllEvents() contract used for migration round-trip). The orchestrator’s _filterByTimestamp does not see camelCase timestamps on those rows and falls to “no timestamp — always include”, which means audit data is currently exported in full on every backup (not incremental). Promoting audit to incremental needs a parallel camelCase export surface and is out of scope for this fix. Regression coverage at [BKP-SHAPE-AUDIT-01..03] pins the missing-await shape so it cannot return silently.
Removed the route handler, the audit.getLogs method + its parameter schema, and every registration site (Paths.Audit, application.ts route wiring, server.ts method wiring, the audit.getLogs entry in the audit ApiMethods list, and the two test-helper registrations). The audit datastore is untouched: recording, the :_audit: store, auditUserStreams/auditUserEvents, syslog, and the storage interfaces all stay — events.get over :_audit: is the supported (and now sole) read path. Audit acceptance tests were repointed to events.get (the deleted legacy-route.test.js is fully covered by the existing [ASTE] events.get suite); Audit.test.js reads audit logs through a dedicated personal “audit reader” access and filters out its own self-audited reads. Dead isAuditActive field in server.ts and dead auditLogs field in Result.ts removed.
storageLayer.events.exportAll returning canonical (camelCase) event objects — the exact shape events.importAll consumes — and the backup/restore orchestrators plus the integrity check go through that store instead of querying engine databases directly. This fixes the PostgreSQL full-backup crash (raw driver result object forwarded as the events array) and the SQLite events black hole (no export, no-op import, and iterateAllEvents/events.clearAll reading an events table in the per-user baseStorage file while live events live in the per-user dataStore file — the teardown integrity pass silently covered zero events). Live round-trip tests cover export/import/clear/iterate on both engines.just clean-test-data resolves PG/rqlite host+port from env overrides, then config/test-config.yml (where parallel checkouts actually carry their port offsets), then canonical defaults — previously an offset checkout reset ANOTHER checkout’s databases while leaving its own dirty. The rqlite full state reset now keys on the local pidfile (any port) and restarts on the same ports; the keyValue-wipe fallback curl gets --max-time 10 (an unresponsive rqlited once stalled the recipe for 40 minutes). rqlite/scripts/start accepts RQLITE_HTTP_PORT/RQLITE_RAFT_PORT.just clean-test-data) instead of a bare “Check should be empty” dump.Dev-dependency refresh; no production dependency changed. mocha 10→11, nyc 15→18, sinon 14→22, superagent 8→10, supertest 6→7. The temp package is replaced by a native os.tmpdir() path in the test-helpers instance manager. The vendored boiler component drops its unused semistandard lint setup (linting is covered by the repo-wide neostandard config). New diff: ^8.0.3 override clears the jsdiff DoS advisory that mocha 11 still pins. npm audit reports 0 vulnerabilities; a fresh npm ci now prints a single deprecation line (glob@10, pinned by latest mocha) instead of fifteen.
Storage-layer and registration robustness fixes, no API impact:
User registration now compensates on failure: the platform reservation (unique/indexed account fields, written first for cross-core uniqueness) plus any partially-created local data are rolled back when the local side of the creation fails, and the original error is rethrown. A failed attempt no longer blocks re-registration of the same unique values nor leaves platform-vs-repository inconsistencies behind.
' by doubling ('') instead of backslash (not an escape character in SQLite) — a value containing a quote produced malformed SQL in the legacy coercion path. Covered by live round-trip tests including injection-shaped values. The streamIds FTS MATCH path was audited and is safe by construction (quotes and backslashes are forbidden streamId characters).CREATE TABLE IF NOT EXISTS and PostgreSQL failed one of them with a pg_type unique-constraint error.idx_webhook_url (per access, live rows only) — duplicate webhook creation returns the documented 409 under both engines, and the collision test is re-enabled.Companion test-infra fixes: the lib-js integration suite now boots HFS correctly (its CommonJS launcher crashed as ESM on every spawn), forwards WebSocket upgrades through the HTTPS test proxy, supports port overrides for parallel local servers, and just clean-test-data fully resets rqlite raft state (stale raft replays previously resurrected ghost platform entries and degraded the local api-server matrix).
A type name now has exactly one meaning across the tree, enforced by lint. just lint fails when a TS source declares a local type/interface whose name collides with a canonical type (the AGENTS.md “Canonical type homes” table) — the error message points at the import instead. Implemented as no-restricted-syntax entries in eslint.ts-any.config.js; only the canonical-home files themselves are exempt. Naming rules: local structural views are XxxLike, engine rows XxxRow, domain-distinct concepts get their own name; when several shapes compete for a bare noun, the API-facing (wire) shape owns it.
A companion ratchet (scripts/open-type-ratchet, chained into just lint) pins the count of open index signatures ([k: string]: unknown) in production sources at 228 — it may only go down. Open index signatures disable typo detection on property reads; new shapes must enumerate their fields unless the openness is load-bearing (dynamic key access, passthrough, the open event data model).
Shipped in two passes, all type-only (no runtime change): first the guard plus three drift fixes (PG/SQLite engine entrypoints import UserOrId instead of carrying identical copies; the mail helper’s Callback renamed MailCallback; five dead type declarations dropped from Result.ts), then a 40-site sweep renaming every remaining bare-noun local (Event→EventLike, Stream→StreamLike or direct StoredStream use, Access→AccessLike/AccessRef, Permission locals replaced by canonical StreamPermission/FeaturePermission imports where identical, StreamQuery→StreamQueryLike/StreamGetOptions, Mall→MallStreamsOnly/MallLike, LogFn→LogLine or boiler imports, ConfigLike→boiler imports, AuditEvent→AuditEventLike, Query→SeriesQuery, plus singles) and a dead duplicate access view removed from the webhooks methods.
DBpostgresql (PG PlatformDB) rewritten from a dormant partial implementation on the pre-2026 platform_unique_fields / platform_indexed_fields tables (never declared in the engine manifest, no production caller) into a complete PlatformDB on a single platform_kv table, key-namespace-identical to the rqlite engine. The legacy two-table DDL is removed from the PG schema; existing deployments keep the (empty, unread) tables until dropped manually.PlatformDB.exportAll() in both engines now exports only the user-unique/ + user-indexed/ namespaces — the previous bare user prefix also matched user-core/ rows, which importAll would corrupt into bogus keys (dormant: no production caller before bin/migrate-platform.js).createMigrationRunner only collects migration capabilities from engines selected for at least one storage type — an unselected rqlite engine would otherwise be probed while no rqlited runs.[PGPF] PlatformDB conformance now instantiates the PG implementation directly (it previously exercised the rqlite-backed barrel instance under the PG matrix); new [PSEL] barrel engine-selection test, [CVPE] boot-guard tests, [S3EF] S3 EventFiles conformance (skips without a reachable endpoint), [CENV] config-to-env round-trip tests.The per-store engine classes (Accesses/Streams/Webhooks/Profile × PG/SQLite) used to extend their bases through an untyped require(), which made the parent type any and silenced all override-compatibility checking — the blind spot the findDeletions shadowing shipped through. The bases are now generic over the stored item shape (BaseStoragePG<T> / BaseStorageSQLite<TItem>), declare implements UserStorage<T>, and every subclass extends through a typed require handle binding its item type (new StoredWebhook in interfaces/_shared/domain.ts). One cross-engine payload fix shipped with it (latent — no caller reads it): AccessesPG.delete in the integrity path now delivers { modifiedCount, integrityRecomputed } like the SQLite twin, instead of the raw recompute result. Type-coverage floor unchanged at 81 (actual 81.97%).
The storage interfaces, their engine implementations and the main callers now exchange real domain types instead of structural bags. All type-level — no runtime behavior changed (two deliberate exceptions below); both engine matrices and CI stayed green throughout.
Contracts (storages/interfaces/**):
_shared/domain.ts: StoredEvent / StoredStream / StoredAccess / StoredPermission / SessionData — the storage-side counterparts of the wire types in business/src/types/public.ts (wire vs stored vs engine-row are deliberately distinct).UserStorage<T extends StoredItem> is generic over the stored item; iterateAll now declares the generator shape both engines actually implement; the mongo-era CollectionInfo.indexes field and the dead insertOne options parameter are gone.Sessions / PasswordResetRequests / UserAccountStorage / EventFiles / UserAuditDatabase / BackupReader+Writer fully typed (migration docs are the mongo-era _id formats, named as such; audit sync/SQLite-vs-async/PG dual-mode modelled on all read methods).Data-access layer: new components/mall/src/types.ts exporting interface Mall / MallEvents / MallStreams (classes declare implements); MethodContext.mall uses it; DataStore.supports strongly typed with StoreSupports.
Callers:
CmcAccessLike / CmcClientData in cmc/src/_types.ts (15 per-module Mall views and 10 access shapes collapsed; OutboundDeps/FetchLike declared once).[key: string]: any; auditIntegrityPayload and authorizationHeader live on the base MethodContext (cross-component contracts); canonical PryvRequest carries the typed context.UsersRepository plumbing fields use the storage contracts; PlatformOperation exported from the platform component.Runtime-visible fixes shipped with the typing (each found by the new types):
mall.events.updateStreamedMany on a multi-store query now throws the underlying error instead of failing with an opaque TypeError at iteration.findDeletionRecords — it shadowed the contract’s findDeletions(deletedSince) with different semantics.Guardrails: just type-coverage floor raised 80 → 81 (baseline 81.77%); explicit : any down to 8 lines in 4 documented clusters (heterogeneous middleware registry, multer file bag, boiler logging internals, migration authoring contract).
Closes the long-running TypeScript type-tightening effort. The earlier drain removed : any file by file with minimal-diff local types; this pass optimizes for consistency across files and locks the result in. All type-level — no runtime change intended; both engine matrices and CI stayed green throughout.
Consistency / dedup (each family one commit):
Logger/LogFn + ConfigLike exported from @pryv/boiler; shared AppLike/PryvRequest in api-server routes/_types.ts; MethodNext/ResultBag in methods/_types.ts — ~120 duplicate local declarations replaced by import type.: Function fields (25) given real signatures; JSDoc brace-types (72) deleted in favor of the TS signatures (descriptions kept).[k: string]: unknown / & Record<…> sites: 47 cargo-culted ones removed (consumers only touch named members); scratchpad pipelines, plugin bags and DB-row dictionaries stay, commented. One real leak surfaced and got a named slot (multer files on events.update params).any (Callback<any>, Record<string, any>, Promise<any> — invisible to the old line metric) drained, including the whole storages/interfaces/** family flipped to unknown equivalents.treeUtils children?: this[] (19 casts deleted), generic fromCallback<T>, modelled mongo-style UpdateData in both engine bases, Platform-typed handles, new exported UserManifest for the backup writer chain.PermissionLevel/AccessType imported from business types; new EventsQueryState (storages/interfaces/_shared/types.ts) and HttpHeaders (business types).! cleanup: assertions dropped where guards already narrow; locals capture narrowing across try/catch and closures; one-line invariant comments at the declaration sites the remaining clusters rely on.Guardrails:
just lint-ts-any (chained into just lint, enforced in CI): @typescript-eslint/no-explicit-any as error over TS sources via the dedicated eslint.ts-any.config.js (separate config on purpose — a files: *.ts block in eslint.config.js would pull every pattern-less neostandard style rule onto TS files). The ~28 surviving any lines (heterogeneous middleware registry, method-pipeline scratchpads, storage-plumbing escape hatches, config getter, winston splat, multer bag, per-backend store handles, polymorphic migration contract) carry justified eslint-disable comments in place.just type-coverage: baseline 80.91 %, --at-least 80 regression floor (type-coverage devDependency).tsconfig flips the strict: true umbrella (all eight granular flags were already on individually; future strict-family flags now arrive by default).Xxx canonical / XxxLike structural-local / XxxRow storage row / Wire-Raw wire shapes; most-precise-type-first, unknown only for genuinely-opaque pass-through and validator inputs; never add compiler-appeasing field initializers (= '', ?? 0) — use field!, field? or an explicit union.Deliberately out of scope (next effort: strongly-typed interface I/O): nominal domain types — the remaining same-name local types (MethodContext, MallLike, AccessLike, Event, SqliteDb, AccessRow, …) are real wire-vs-stored-vs-engine-row divergences to be modelled once, then threaded through interfaces → engines → callers.
The dns-active preflight’s delegation check did a recursive NS lookup, which needs the zone’s own authoritative server to answer. At wizard time that server is the not-yet-booted embedded DNS, so the check could only fail and warned “No NS delegation found” on every correctly-delegated first boot — training operators to ignore the one check that catches real delegation mistakes.
The check now walks up the label chain to the nearest ancestor zone a public recursor can resolve, then queries one of that zone’s nameservers directly for NS <domain> (via dns2, now an explicit root dependency; 3s timeout race since the bare UDP client has none). Parents serve delegation referrals from their own zone data, so the answer is independent of the child zone’s server being up. Three outcomes: delegation found (✓ + the NS→A-vs-publicIp cross-check as before), parent explicitly serves no delegation (warning — now a true config-error signal), lookup itself failed (informational note, best-effort tier). Verified live against a real parent zone in all three states: correct delegation pre-boot (no warning), undelegated name (warning), delegation pointing at a different IP than declared (wrong-server warning).
registerSelf falls back to dns.publicIpIn dns-active (subdomain-per-user) mode the embedded DNS answers the distribution-reserved service names (reg.<domain> / access.<domain> / mfa.<domain>) and <coreId>.<domain> from the ip field of each core’s PlatformDB self-registration. Platform.registerSelf() only read core.ip — a key that neither the config wizard nor the bootstrap bundle sets (both carry dns.publicIp) — so those rows landed IP-less and the names returned empty answers, while /reg/service/info kept advertising register: https://reg.<domain>/: standard onboarding against the advertised register URL could not resolve. Verified live on a wizard-generated single-core deploy (LE staging, real public resolvers).
Fix: registerSelf() now falls back to dns.publicIp for CoreInfo.ip when dns.active is on and core.ip is unset; an explicit core.ip still wins, and non-dns-active deployments are unchanged. 3 new [MCIP] tests pin the fallback, the precedence, and the dns-inactive no-op.
rqliteProcess.buildArgs() passed -http-addr 0.0.0.0:<port> unconditionally, while only the raft listener honored the single-core-stays-on-loopback rule the surrounding comment promised. The rqlite HTTP API is plaintext and unauthenticated with full read/write access to PlatformDB (DNS records, core registry, invitation tokens, access-state, encrypted TLS-cert blobs); docker deployments were shielded by the port mapping, but raw single-core hosts exposed it to anything the host firewall let through. Single-core (no core.ip) now binds 127.0.0.1:<port> — all in-process and on-host clients connect via localhost, so nothing changes for them. Multi-core keeps 0.0.0.0 + -http-adv-addr <core.ip> (NAT-aware, as before). The dev/test launcher storages/engines/rqlite/scripts/start carried the same 0.0.0.0 literal and is fixed to loopback too (observed live: a long-running dev rqlited answering unauthenticated on a dev machine’s public interface). [RQARGS] argv tests updated + a dedicated loopback assertion added.
Closes the out-of-the-box gap for non-dnsLess (subdomain-per-user) single-core installs. Previously the embedded DNS server shipped empty, so even though ACME published the DNS-01 TXT in-memory, public recursors saw no authoritative answer for the zone and acme-client errored at preflight (No TXT records found for name: _acme-challenge.<domain>). Operators had to hand-run bin/dns-records.js load with a SOA/NS/A YAML before first boot — undocumented and easy to miss.
bin/master.js seeds the apex DNS chain at startup when dns.active: true + dns.domain set. Before workers fork + the ACME orchestrator starts, it merges a bootstrap SOA + NS (→ core.<domain>) into dns.records.root in-memory and publishes an A core.<domain> → dns.publicIp record via PlatformDB. Idempotent: skips whichever records already exist, so operator-edited records always win. If dns.publicIp is unset, it logs a FATAL banner naming the key and skips ACME startup (rather than burning the LE rate limit on a guaranteed-fail issuance).dns.publicIp config field (under the dns: block) — the host’s public IPv4, consumed by the bootstrap above. Distinct from core.ip (multi-core CoreInfo).dns.publicIp on the non-dnsLess path, auto-detecting a default via checkip.amazonaws.com (2.5s timeout, falls back to manual entry).dns module — no dig dependency) to check (1) whether the parent zone delegates dns.domain at all, (2) whether the delegated NS hostnames resolve to the declared dns.publicIp (mismatch → wrong-server warning), and (3) a UDP/53 reachability probe to dns.publicIp (informational — silence is expected before first boot; an answer flags an existing listener / possible host systemd-resolved conflict). Surfaced in the existing “Validating host environment” summary.AcmeOrchestrator arms the renew timer at 60s when no cert is stored yet, downshifting to the normal 24h interval the moment a cert is present (first success OR restart with an already-issued cert). A #renewInFlight single-flight guard prevents the start()-side immediate trigger and the 60s timer from issuing concurrently. This absorbs the common first-deploy symptom where LE’s distributed validators don’t see a freshly-published TXT within the initial window (negative-cache hangover from the pre-delegation period) — the next 60s attempt succeeds instead of waiting a full day. PlatformDBDnsWriter default waitMs also bumped 15s → 30s for extra public-recursor cache headroom.bin/dns-records.js --help notes that operator-managed records now coexist with the master-auto-published bootstrap chain (master only writes records that don’t already exist). INSTALL.md gains a “first-boot DNS chain (dns-active mode)” subsection.Validated end-to-end on a real host with NS delegation: real LE production wildcard issued on first boot (after the 60s retry), cert restored from disk on restart, both confirmed externally via openssl s_client.
First Release Candidate divider. All entries below this line up to the next ## 2.0.0-* divider were landed during the 2.0.0-pre rolling line and are now sealed under the RC tag. Internal changes are no-API-impact; see CHANGELOG-v2.md for implementer-facing changes.
Notable internal work since 2.0.0-pre open: SQLite full-matrix parity (Plan 76 close, both engines at 2351/0/7), TypeScript + ESM migration foundation (Plan 57 + Plan 64 strict-mode + Plan 65 noImplicitAny + ongoing Plan 80 type tightening), test-isolation hardening ([P4OM] customAuthStepFn race fixed), docker publish gated on git tags only.
bin/init.js’s defaultImageRef() reads process.env.PRYV_IMAGE_TAG (baked at image build time via ARG IMAGE_TAG → ENV PRYV_IMAGE_TAG in the Dockerfile) to compute the default pryvio/open-pryv.io:<tag> literal in the generated run-pryv.sh + check-config.sh launchers + the “Start the server” / “Verify the config” fallback hints. CI’s docker/build-push-action@v7 step passes --build-arg IMAGE_TAG=$ on tag pushes, so the published image always carries the same tag it was published under.
Local docker build . without --build-arg IMAGE_TAG=... falls back to dev — a tag that doesn’t exist on Docker Hub, so a wizard run from a local build surfaces the misconfiguration cleanly when the operator runs ./run-pryv.sh and docker can’t pull. No more stale :2.0.0-rc.1 literals across RC bumps.
Two adjacent bugs in the embedded ACME flow that produced the same operator-visible symptom (workers serving a 1-day self-signed cert even after Let’s Encrypt successfully issued the real one):
First-boot hot-swap silently re-loaded the placeholder. The onRotate IPC fanout in bin/master.js sent {type:'acme:rotate', certPath, keyPath} with the FileMaterializer’s paths (<tlsDir>/<hostname>/{fullchain,privkey}.pem), but workers’ Server.reloadTls() ignores the message paths and re-reads from http.ssl.{certFile,keyFile} (the configured boot-time paths) — which still held the placeholder. Fix: master copies the rotated cert over http.ssl.{certFile,keyFile} BEFORE the IPC fanout. New log line [acme] copied rotated cert <materializer-path> -> <ssl-file> precedes the worker reload.
Container restart re-overwrote with a stale placeholder. selfSignedPlaceholder.ensure() checked whether the configured ssl files existed and short-circuited “cert-files-already-exist” — leaving the stale placeholder in place. Even though FileMaterializer had a real LE cert at <tlsDir>/<hostname>/, the placeholder code never looked there. Fix: reorder the function to check the materializer’s per-hostname dir FIRST. If a real cert is present, copy it over http.ssl.{certFile,keyFile} before workers fork. New return field { restored: true, source: <materializer-path> } + log line [acme] restored materialized LE cert for <host> from <materializer-path> -> <ssl-file> on the restart path.
The wizard now pins letsEncrypt.tlsDir: <install-dir>/data/tls so the materializer’s per-hostname dir lives on the same operator-mounted persistent volume as http.ssl.* — both paths converge cleanly on container restart. Operators with custom tlsDir paths get the same behaviour as long as the directory is persisted across container removal.
3 new unit tests pin the behaviour: [SS20] (dnsLess + http-01 restore), [SS21] (dns-01 wildcard restore — wildcard.<host> dir name), [SS22] (half-materialized state — only fullchain.pem present — falls through to placeholder rather than serving an incomplete cert).
/app/pryv mount, sectioned YAMLbin/init.js reshaped for the docker run -v "$(pwd):/app/pryv" pryvio init UX. Concrete changes:
/app/pryv. Host path discovered via /proc/self/mountinfo. Local-dev escape hatch: PRYV_CONFIG_DIR=/tmp/x node bin/init.js.pryv-config.yml (was: operator-named). Run-pryv.sh + check-config.sh launchers reference this filename verbatim, so the in-container --config arg in the generated docker run matches the on-disk filename without operator action.<install-dir>/data (sibling to the config). One operator-mounted volume covers both. The data-folder prompt is dropped.letsEncrypt.tlsDir pinned. <dataFolder>/tls, on the same persistent mount as http.ssl.{certFile,keyFile} — see the ACME fix above.check-config.sh companion launcher. Mirrors run-pryv.sh’s self-locating + overwrite-on-exists pattern. Validates without booting.pryv-config.yml now opens with a banner + # ── Section ─── dividers + per-section docstring comments. Logical top-down read order: identity → topology → HTTP+TLS → LE → auth → web-auth3 → cluster → storage. Ends with a commented-out optional-sections appendix (services.email, services.mfa, hostings, custom.systemStreams, observability, eventFiles, webhooks, cluster.discoveryEnabled, core.url).run-pryv.sh + check-config.sh are created unconditionally on a fresh install, asks before overwriting an existing one. “Next steps” hints quote ./run-pryv.sh + ./check-config.sh (relative); fallback docker-run snippets use $(pwd) for the host-path source so the snippet pastes correctly from any cwd.Maintenance directive: the optional-sections appendix in buildOptionalAppendix() must stay in sync with config/default-config.yml + config/production-config.yml. New top-level sections an operator commonly tunes need a mirror edit there.
The docker job in .github/workflows/ci.yml now publishes
pryvio/open-pryv.io to Docker Hub only on git tag pushes, not on
every push to master. Master receives many no-behaviour-change commits
(TS tightening, lint, comment edits) and publishing an image per
commit churned the registry without any release-meaningful artefact.
The release flow becomes:
git tag -a 2.0.0-pre.5 -m "..."
git push --tags
Published tags:
pryvio/open-pryv.io:<git-tag> — the named release (e.g. 2.0.0-pre.5)pryvio/open-pryv.io:2.0.0-pre — rolling pointer, updated to the
latest tag so hosts already pulling :2.0.0-pre keep tracking the
current release candidateThe per-sha tag pryvio/open-pryv.io:2.0.0-pre-<sha> has been
retired. Pin hosts to a named tag instead.
just test-sqlite all now matches just test all (PG) at exit=0,
0 fail, same baseline pass count. Six fixes, each gated narrowly so
the PG matrix is untouched:
test-helpers/src/helpers-base.ts module-load engine
override. STORAGE_ENGINE=sqlite now propagates to every test
component that loads helpers-base.ts directly (audit, cache,
mall, webhooks, …), not just to api-server which loads
helpers-c.ts. Without this, non-api-server components booted on
the PG default and saw Pattern A child cores’ cross-engine
writes in their first checkIndexAndPlatformIntegrity hook.
storages:audit:engine is DELIBERATELY left at the default —
PG audit storage (UserAuditDatabasePG.createEvent) has a
pre-existing eventid NOT NULL violation that fires on
[ASTO] if audit is routed through PG. Override gated to
STORAGE_ENGINE === 'sqlite': under PG the memory-scope set
blocked later injectTestConfig resets the mall suite relies
on and timed [MS04]/[MS08] out.
test-helpers/src/helpers-base.ts matrix-hygiene wipe in
mochaHooks.beforeAll — calls platform.deleteAll() +
usersLocalIndex.deleteAll() once per non-api-server component
process. Clears leftover users that api-server’s
versioning.test.js [VE07] (POST /users with no cleanup)
writes to the persistent rqlite + per-user-file SQLite index.
Without this, mall [2Z7L] and audit [U2PV] saw 4-vs-1
repo-vs-platform drift in their per-test integrity hook. Gated
to SQLite + non-api-server (api-server runs first and racing
with helpers-c.ts dependencies.init() regresses
[ACUP07]/[EVNT]).
TestServerContext.spawn engine pass-through (SQLite only)
— copies parent’s effective storages.{base,series,file}.engine
into the spawned child’s injectSettings. Fixes the 3-failure
webhooks [WH01] block: the test parent spawns an api-server
child via context.spawn(...), and that child was booting with
the default engine. The test wrote a user to the parent’s
SQLite store; the child looked it up in PG → 404. Settings
are passed via injectSettings (test scope) rather than env
(memory scope) so non-SQLite consumers can still pin their
child to PG.
SQLite engine: wire the cache internal end-to-end. The
engine manifest gains cache in requiredInternals;
_internals.ts exposes the getter;
StreamsSQLite.{insertOne, updateOne, delete} call
_internals.cache.unsetUserData(userId) (or unsetStreams for
non-structural updates) to invalidate the per-user
permission-level cache on writes — mirroring PG StreamsPG.
localUserStreamsSQLite._getAllFromAccountAndCache reads /
sets the streams cache so subsequent mall.streams.get is a
cache hit. Closes cache [XDP6]: under SQLite, stale
AccessLogic permissions after streams.update reparenting
kept granting access to a parentId: null move-out.
cache/test/acceptance/cache.test.js [FELT] SQLite skip
on the 15%-speedup timing-gain assertion. Local SQLite’s
per-user-file fetch is fast enough that the cache memoization
adds more overhead than it saves; the isFull() cache-hit
invariant earlier in the same test still verifies the
integration. Same skip already existed for CI for the same
reason. (A future cache-evaluation pass can revisit whether
local memoization is worth keeping when storage round-trips
are sub-millisecond.)
api-server/test/helpers/validation.js checkObjectEquality
— always exclude integrity from comparison when the
expected object doesn’t carry one. Previously this only
skipped under “approximate match” (i.e. when
actual.modified !== expected.modified); under PG the times
always drifted from the test’s post-response
timestamp.now() so integrity got skipped, but under SQLite
the modified timestamp could match exactly, integrity got
compared, and the test failed with a phantom
+ integrity: EVENT:0:sha256-... diff. Removes the
engine-shape coupling from [4QRU] (events PUT),
[AC01]/[AC18] (accesses), and the cluster of
[EVNT]/[CMCNS] intermittent fails the SessionState had
tracked as “matrix-only test isolation”.
SpawnContext prespawned a pool of child processes at module-load
time, capturing process.env then. Under MOCHA_PARALLEL=1 the
per-worker DB names + rqlite URL injected by setupParallelWorker in
the parent’s mochaHooks.beforeAll arrived AFTER prespawn — so the
prespawned children booted against the default pryv-node-test DB and
the wrong rqlite endpoint. Tests that grabbed a prespawned child then
failed on stale connections (the [SDHF] Pattern A cold-start
failures in hfs-server/test/acceptance/store_data.test.js).
TestServerContext (new file
components/test-helpers/src/TestServerContext.ts) is a drop-in
replacement with the same external API: spawn(customSettings?),
shutdown(), Server exposing request() / baseUrl /
url(path) / stop() / process.sendToChild(cmd, ...args). The two
material changes:
spawn() call, never
ahead of time. No prespawn pool to drain or refill.process.env is snapshotted at fork
time, so per-worker config injected after module load reaches the
child.IPC protocol with the child is unchanged (msgpack [msgId, cmd,
...args] → child ChildProcess handler → ['ok'|'err', msgId, cmd,
ret|errJson]), so the existing launcher scripts
api-server/test/helpers/child_process.js and
hfs-server/test/support/child_process.js are untouched and
server.process.sendToChild('mockAuthentication', false) style mock
injection still works.
Consumers migrated:
api-server/test/test-helpers.js — context = new TestServerContext()hfs-server/test/acceptance/test-helpers.js — spawnContext = new TestServerContext('test/support/child_process')api-server/test/helpers/index.js — re-export shifted from
SpawnContext + InstanceManager to TestServerContext. Legacy
InstanceManager (a sibling of DynamicInstanceManager that no
test file actually consumed directly) is also dropped from this
surface.The legacy spawner.ts (367 LOC) and InstanceManager.ts (180 LOC)
are deleted. DynamicInstanceManager (used by api-server +
previews-server dependencies.instanceManager for the modern
in-process startup path) is unaffected.
Verification: just test hfs-server 60/0, just test webhooks 8/0
sequential PG. just test-parallel hfs-server 60/0 — closes the 4
[SDHF] [SD01]/[SD02] parallel-mode failures. Full
just test-parallel all: ~5 failing (down from ~10), all remaining
failures are pre-existing parallel-mode flakes in audit
(log-count race) + api-server ([PFRC], [PCRO]) — neither uses
the spawn surface.
Code-meta-cleanup pass on tests, comments, and component READMEs. No production behavior change.
[H2GC] (result.test.js
never-implemented stub) and [APRA] (accesses.test.js Pattern-A→C
migration leftover).differnt→different for [ZUTR], doubled error
for [60OQ]).TODOs walked: most deleted as aspirational, the few
flagging real follow-ups rewritten as dated entries anchored to the
internal bug log. Three surfaced new bugs (hfs-server audit-context
literal 'TODO' IP, events.get forced/forbidden-id dedup,
test-helpers/spawner port leak) which are now triaged.test-helpers/src/child_process.ts,
api-server/test/acceptance/account.default-streams.test.js).components/audit/README.md, components/storage/README.md).README-DBs.md
(userLocalDirectory.js→.ts,
userLocalIndex.js→usersLocalIndex.ts).Test matrix unchanged: just test api-server 1073 passing / 0 failing
/ 6 pending; just test business 384 passing / 0 failing.
accesses.delete did not remove webhooks attached to the deleted
access (or its descendant shared accesses for an app-access
deletion). Webhook rows survived with a dangling accessId and kept
firing notifications until manually cleaned. The data exposure is
bounded by the signal-only design (the receiver’s GET back authenticates
with the now-deleted access and gets 401), but the outbound channel
itself was not torn down.
Fix is three small additions:
Repository.deleteByAccess(user, accessId) in
components/business/src/webhooks/repository.ts mirrors the existing
deleteOne / deleteForUser shape.deleteAccesses middleware in
components/api-server/src/methods/accesses.ts walks the same
idsToDelete list (app access + descendants) and calls
webhooksRepository.deleteByAccess for each — before the access
row is deleted, so a partial failure leaves a retryable state.Webhook.send() fire-time access-validity check: on cache miss for
the parent access, the repository’s new accessExists(user, accessId)
is consulted and the webhook self-deactivates (state = 'inactive')
when the access is missing or tombstoned. Self-heals orphan webhooks
from before this fix shipped, and any future code path that creates a
dangling webhook.WebhooksRepository constructor gained an optional third parameter
accessesStorage; existing two-arg callers still work (defensive
default accessExists returns true, so no-op for legacy wiring). Three
known instantiation sites updated: api-server/src/methods/webhooks.ts,
api-server/src/methods/accesses.ts (new), and
webhooks/src/service.ts (the dispatcher that actually fires webhooks
in-process per Plan 14).
Tests:
[WCAD] 3 tests in components/api-server/test/acceptance/accesses.test.js
cover the cascade (own webhook, descendant webhook, unrelated webhook
untouched).[WCAD-FIRE] 3 tests in
components/business/test/acceptance/webhooks/Webhook.test.js cover
the fire-time check (missing access, tombstoned access, live access).Closes the bug chips on hipaa-security.164.308(a)(3)(ii)(C),
iso-27001.A.5.16, iso-27001.A.5.18 in compliance-matrix/.
GitHub: pryv/open-pryv.io#82.
audit.onUserDelete (Plan 72 A.2)Layered on top of A.1 (which made the erasure engine-consistent), A.2 surfaces the policy choice operators need for retention regimes that conflict with the GDPR Art.17 default:
erase (default) — auditStorage.deleteUser(userId) runs (A.1 path). Matches GDPR Art.17 / CCPA §1798.105 / PIPEDA Principle 4.5 default.keep — deleteAuditDataStorage middleware skips the wipe with an info-log naming the user. For HIPAA §164.316(b)(2)(i) (6-year audit retention regardless of subject erasure), MDR Art.10(8) (10-year device-history retention), or any regime keeping the audit under a separate lawful basis (GDPR Art.17(3)(b) — compliance with a legal obligation). The operator must document the retention in their DPIA.pseudonymise — REFUSED AT BOOT. Depends on the not-yet-shipped auth.randomAlias primitive (open-pryv.io#38, backlog slug ALIASES). config/plugins/config-validation.js refuses this value with a clear error message naming the dependency, so operators selecting this mode get a deterministic failure instead of a silent fallback. If somehow seen at runtime (override during a hot reload), the middleware logs a warning and falls back to erase defensively.Wiring:
config/default-config.yml audit.onUserDelete: erase (with multi-line comment documenting all three modes + their lawful-basis fit).config/plugins/config-validation.js gains checkAuditOnUserDeleteMode(config, problems) — enum-style validation + the pseudonymise gate. New exports: checkAuditOnUserDeleteMode, AUDIT_ON_USER_DELETE_MODES.components/business/src/auth/deletion.ts::deleteAuditDataStorage reads audit:onUserDelete (defaults to erase if absent) and branches before the existing A.1 wipe.Tests:
[CV-AOUD] 6 unit tests in components/api-server/test/config-validation-required-when-seq.test.js covering: erase/keep accepted, pseudonymise refused with ALIASES dependency mention, unknown enum rejected with allowed list, absent value tolerated, enum exposed.Known caveat — SQLite audit + keep mode: the parent deleteAuditData step (filesystem wipe of userLocalDirectory) still runs after deleteAuditDataStorage returns from keep. For SQLite-audit deployments, the per-user .sqlite file lives inside that directory and gets wiped regardless. PG-audit deployments work as designed (rows survive). Operators wanting keep semantics on SQLite need either a PG audit migration, or a follow-up that teaches deleteAuditData to skip when onUserDelete === 'keep'. Logged for follow-up; not blocking the chip discharge.
Closes 3 companion feature chips on the compliance-matrix (gdpr.Art.17 feature, ccpa.1798.105 feature, hipaa-security.164.316(b)(2)(i) feature). GitHub: pryv/open-pryv.io#75 (same issue as A.1; both phases discharge together on merge).
auth.delete (the GDPR Art.17 erasure primitive) silently left PG audit_events rows referencing the deleted subject behind. The existing deleteAuditData step in business/src/auth/deletion.ts only wiped the per-user filesystem directory — sufficient for SQLite (whose audit DB is a file inside that tree), insufficient for PG (whose audit_events is a shared table keyed by user_id). AuditStoragePG.deleteUser existed but had only one in-tree caller (the backup-restore preflight); the auth.delete pipeline did not invoke it.
Fix:
deleteAuditDataStorage middleware in components/business/src/auth/deletion.ts calls require('storages').auditStorage?.deleteUser(context.user.id) (null-guarded — defensive for any future engine that doesn’t declare auditStorage in its manifest; today PG and SQLite both do).auth.delete pipeline in components/api-server/src/methods/auth/delete.ts BEFORE the existing deleteAuditData filesystem wipe — so the SQLite path closes the per-user DB file cleanly before the directory rm fires (avoiding forUser re-creating the dir during the close).[USAD][9] test in components/api-server/test/deletion-seq.test.js (“should delete user audit events”) gains an engine-agnostic assertion using auditStorage.forUser(userId).countEvents() === 0. The old fs.existsSync regression-guard for the SQLite path stays.Closes the bug chips on gdpr.Art.17, ccpa.1798.105, iso-27701.A.7.4.5 in compliance-matrix/. GitHub: pryv/open-pryv.io#75. Three companion feature chips on the same rows (operator setting audit.onUserDelete: erase|keep|pseudonymise) stay until Plan 72 Phase A.2 ships.
defaultName/name extrasCloses a long-standing wire-format asymmetry between accesses.create, accesses.checkApp (returns checkedPermissions shaped per CREATE), and accesses.update: the first two accepted/produced permission objects with optional defaultName + name fields used during app-authorization UI; the third strictly rejected those fields with OBJECT_ADDITIONAL_PROPERTIES. Naive callers piping checkApp.checkedPermissions straight into accesses.update hit invalid-parameters-format; HDS worked around it by stripping extras client-side in bridgeAccess.ts / Authorization.tsx (see workspace BUGS B-2026-05-14-4).
components/api-server/src/schema/access.ts permissions(action) now extends the streamPermission shape with defaultName + name for Action.UPDATE as well as Action.CREATE — wire-format-symmetric. A new cleanupUpdatePermissions middleware (mirror of the existing cleanupPermissions used on CREATE) strips those fields in accesses.update before snapshotAndApplyUpdate persists, so on-disk shape is unchanged. 2 new [ACUP-SYM] tests: [SYM01] PUT with defaultName+name returns 200 and stored permission has neither field; [SYM02] full checkApp → accesses.update round-trip with checkedPermissions sent verbatim returns 200. The pre-existing [PA03] CMC-handshake test comment that documented the workaround is updated to point at the fix.
Behaviour on the wire is strictly additive — pre-fix callers (who sent bare {streamId, level}) keep working unchanged; the change just stops rejecting the wider shape.
items.filter is not a functionBackupOrchestrator._filterByTimestamp previously called items.filter(...) directly; if any of the 6 upstream exportAll/exportAllEvents calls (streams / accesses / profile / webhooks / events / audit) ever returned a non-array (shape drift in a storage-layer wrapper, missing collection for a partially-provisioned user, etc.), the backup crashed at the first such call with TypeError: items.filter is not a function and no hint about which collection produced bad shape — see workspace BUGS B-2026-05-20-1 (HDS hit this in prod on the very first user). Without prod data to repro locally, the root-cause shape mismatch can’t be triaged from this side; the value here is making the next occurrence diagnose itself.
_filterByTimestamp now takes a source label (streams/accesses/webhooks/events/audit) and asserts Array.isArray(items) before filtering — non-arrays throw a clear error: Backup export shape mismatch: expected array from "streams" (user <id>), got object keys=[rows]. Likely a storage-layer return-shape drift; .... The single profile export (no timestamp filtering) gets the same guard via a sibling _assertArray. 7 new [BKP-SHAPE-01..07] unit tests in components/business/test/unit/backup/orchestrator-shape-guard.test.js. Smoke: just test business 381/0 (was 374 + 7 new tests).
DELETE /system/users/:username error message self-documentingThe admin endpoint refuses calls without ?onlyReg=true because it only deletes the user’s platform-side fields (uniqueFields like email + indexedFields) — it does NOT cascade through base storage (events, streams, attachments) or the audit log. Previous error text “This method needs onlyReg=true for now (query)” said nothing about why — operators ran into it, assumed the endpoint was broken, then either gave up (orphan test users on shared hosts) or filed bug reports. New error text spells out the partial-delete semantic + points at ?dryRun=true for preview. Closes B-2026-05-14-5 (workspace BUGS.md). Endpoint behaviour unchanged. Existing [GF30]–[GF33] tests still green (6/0 in npx mocha --grep GF3).
platformStorage from postgresql/mongodb manifests + fail fast on engine/storageType misdeclarationPlan 25 made rqlite the only platform engine in production: default-config.yml ships storages.platform.engine: rqlite, and the PG / Mongo PlatformDB implementations are intentionally incomplete (missing the Plan-27 DNS-records methods, the Plan-55 access-state methods, the Plan-35 LE TLS-cert + ACME-account methods, the Plan-38 observability-secrets methods — see workspace BUGS B-2026-05-21-1). Their manifests nevertheless declared "platformStorage", which made pluginLoader.getEngineFor('platformStorage') happily return postgresql when a test config quirk set it that way — only to fail much later via the validatePlatformDB interface validator with a cryptic PlatformDB implementation missing method: <X> and no hint that the root cause was the engine selection.
Two-part fix:
storages/engines/postgresql/manifest.json and storages/engines/mongodb/manifest.json no longer declare "platformStorage". Their createPlatformDB exports remain (legacy, unused) but the manifest now matches the Plan-25 reality.Loader fail-fast — storages/pluginLoader.ts resolveConfig now throws when a configured engine doesn’t declare the requested storageType, e.g.:
Configured engine "postgresql" for storages.platform.engine but engine
manifest does not declare storageType "platformStorage". Engines
declaring "platformStorage": rqlite
This surfaces a misconfigured engine selection at init time with a clear message, instead of letting it fall through to the interface validator’s missing-method error.
tools/coverage/pg-early-init.js had platform: { engine: 'postgresql' } left over from before Plan 25 / before the PlatformDB scope grew — corrected to rqlite.
New test [PLUG-RESOLVE-MISDECLARE] in storages/test/pluginLoader.test.js (48 passing). Component smoke tests just test business (374/0), just test mall (22/0), just test storages (48/0) all green.
clean-test-data falls back to system dropdb/createdb/mongosh when local var-pryv/<engine>-bin/ is absentThe clean-test-data + clean-test-data-parallel recipes hardcoded ./var-pryv/postgresql-bin/bin/dropdb etc. — if that local install was absent (e.g. on a Darwin dev box where the operator uses Homebrew / Postgres.app instead of running storages/engines/postgresql/scripts/setup), the recipes swallowed the binary-not-found failure as ... not reachable (skipping ...) and silently left the prior run’s PG/Mongo state in place. Tests then tripped on stale residue (e.g. webhook-lifecycle [WH01] flakes documented in workspace memory).
Both recipes now resolve DROPDB/CREATEDB/MONGOSH by preferring the local Plan-41 install if present, otherwise falling back to whatever is on PATH (command -v dropdb etc.). The skip-message path is preserved but now distinguishes “binary not found” from “server not reachable” so operators see the real cause.
Verified on Darwin: local bin present → unchanged behaviour; local bin hidden + system bin on PATH → fallback invoked correctly; both missing → clear “not found” message instead of misleading “not reachable”.
accesses.create / accesses.updateThe 5 reserved parents under :_cmc:* are pre-provisioned at user creation by components/cmc/src/provisioning.ts. Per-app sub-trees under :_cmc:apps:<app-code> were historically created on-demand at CMC-acceptance time (provisioning.ts:21-26) — but the OAuth-grant flow used by doctor-dashboard (via app-web-auth-3) never reaches an acceptance event before the first invite, leaving the per-app root :_cmc:apps:<app-code> missing. Downstream streams.create for a child of the leaf then failed with unknown-referenced-resource (“Unknown referenced unknown Stream”). Bridge-onboarded doctors escape this because their onboarding flow uses a personal token to pre-create the stream — OAuth-onboarded ones cannot.
New createAccessProvisionAppScopeHook in components/cmc/src/hooks.ts, exported via index.ts, wired post-createAccess in accesses.create and post-snapshotAndApplyUpdate in accesses.update (components/api-server/src/methods/accesses.ts). Scans result.access.permissions for any streamId resolving via C.getAppCode() to a valid app-code (matches /^[a-z0-9-]+$/, excludes reserved chats/collectors segments), and lazy-creates the leaf :_cmc:apps:<app-code> as a child of :_cmc:apps via mall.streams.create — same bypass pattern as provisionUserStreams so the reserved-root hook doesn’t reject our own provisioning. Provisioning failures are logged but don’t fail the access response (the access is already stored; surfacing here would confuse the caller — if the stream truly can’t be created, the user’s first child streams.create will surface the same downstream error). Deep app sub-trees (:_cmc:apps:<app>:chats:* / :_cmc:apps:<app>:<...>:collectors:*) keep their on-demand-at-acceptance-time behaviour — this hook only provisions the leaf root.
NEW [CMCHS-AP-PER-APP] describe in components/api-server/test/cmc-handshake.test.js (4 tests, positioned after the existing [CMCHS-AP] block from cad7627):
[PA01] — accesses.create with a :_cmc:apps:<new-app> perm auto-provisions the leaf (verified by re-attempting the leaf create and asserting item-already-exists).[PA02] — accesses.create referencing a pre-existing leaf perm returns 201 (idempotent on the provisioning side).[PA03] — accesses.update that adds a per-app perm provisions the new leaf. (Send bare {level, streamId} perms in the update body — accesses.update rejects the lenient accesses.create shape per the documented permissions-shape asymmetry.)[PA04] — accesses.create with a deep :_cmc:apps:<app>:chats:* perm also provisions the leaf, since any descendant create requires it.just test cmc 396/0; just test api-server 1060/2 (the 2 failures are pre-existing [WH01] webhook lifecycle flakes from stale DB residue, documented in workspace memory, unrelated to this change).
Closes a long-running internal effort to make just test-parallel all (the local-dev productivity tool) survive 14-worker concurrency on a 15-core dev box. Matrix went from a broken 1654/79 baseline (with ~480 tests silently hidden) to 2248/68/3 in ~2:06 wall — and the remaining 3 failures are Pattern A cold-start flakes that don’t reproduce at CI’s 2-worker scale. CI continues to run the sequential PG matrix (just clean-test-data && just test all) as the matrix-of-record because parallel mode disables integrity checks, the caching layer, and cluster_kv IPC fallback semantics — three production-relevant verifications that the sequential mode exercises.
Key fixes that landed:
clean-test-data-parallel auto-derives WORKERS from MOCHA_JOBS / cpus-1 (was hardcoded '8', hiding workers w8..w13 on 15-core boxes).computePgPoolMax() in parallelWorkerSetup.ts + bumped var-pryv/postgresql-data/postgresql.conf max_connections 50 → 300; also bumped in fresh-install storages/engines/postgresql/scripts/setup.DynamicInstanceManager.start() force-pins per-worker DB names into the spawned api-server’s tempfile config, defending against mid-suite boiler-config reverts.mochaHooks coverage extended to webhooks/, previews-server/, storages/engines/postgresql/. New storages/engines/rqlite/.mocharc.cjs so engine tests don’t inherit a non-applicable test/hook.js.business/ mocharc + hook now chain helpers.dependencies.init() into beforeAll so every parallel worker initialises the StorageLayer proxies — fixes the [WHBK] Repository.insertOne hang.[XS12] accessState clear routed through cluster.request(0, ‘clear’) because parent test process doesn’t init storages.[ACMEINT] acme-integration.test.js now uses per-worker rqlite URL (via boiler env-mirror) — eliminates the worker-0 leader-election 503 race.setupParallelWorker pre-inits the per-worker rqlite keyValue table (closes [PCRO]/[RGLG] “no such table” race).beforeEach/afterEach (sequential mode) now gated on storages.storageLayer being initialised — closes the historical Darwin Mongo [EVNT] crash where ensureBarrel() early-init locked pluginLoader to the wrong engine before injectTestConfig could apply the override.api-server/test/support/httpServer.js and business/test/acceptance/webhooks/support/httpServer.js resolved by per-worker port shift (6123 + MOCHA_WORKER_ID*10) + a proper await listen() Promise wrapping 'listening'/'error'.Remaining flakes (3 tests in api-server: [ACCO]×2, [SYRO]) + 4 deferred follow-ups (hfs-server SpawnContext migration, port-collision proper fix shifting RQLITE_HTTP_BASE 4001→4011, harness cleanup, two non-api-server -seq files) tracked outside this repo for a future pass. A new test-parallel-all.sh wrapper (in the orchestration workspace) runs pre-checks (stops host rqlited, ensures PG + InfluxDB up) then clean-test-data-parallel + test-parallel all.
:_cmc:* permissionsImplementation detail for the user-visible behaviour documented in CHANGELOG-v2.md. accesses.ts::createDataStructureFromPermissions::ensureStream now early-returns when permission.streamId.startsWith(':_cmc:') — the local-store streamId-validity regex (^[a-z0-9-]{1,100}) was rejecting valid CMC-plugin stream-ids like :_cmc:inbox and :_cmc:apps:<app>. The skip is intentionally narrow: the existing :_system: / :system: path is untouched (parses to account store, not local), and any non-CMC local streamId with forbidden characters is still rejected with the same invalid-request-structure error.
Also fixes the chartacter → character typo at the three sites that share the error wording (accesses.ts, helpers/commonFunctions.ts, helpers/streamsQueryUtils.ts).
NEW [CMCHS-AP] describe in components/api-server/test/cmc-handshake.test.js (3 tests):
[AP01] — accesses.create with :_cmc:apps:<app> + :_cmc:inbox perms returns 201 + preserves perms.[AP02] — full doctor-dashboard / app-web-auth-3 shape (local app stream + two :_cmc:* perms in one call).[AP03] — regression-pin: truly invalid local streamIds still get rejected with invalid-request-structure + the fixed forbidden character(s) message.Positioned LAST in the file for the same ordering reason CN14 is — extra alice-side :_cmc:* accesses confuse the (username, host, appCode)-keyed back-channel matcher used by CMCHS-IDEMP / CMCHS-EXT / CMCHS-SU.
Matrix at close: just test api-server (PG) 1058/0/7.
storages/engines/influxdb/scripts/setup rewritten to mirror the PG / rqlite pattern — self-contained, no system packages, no sudo. Pins InfluxDB 1.8.10 (matches the influx 1.x npm client; 2.x is API-incompatible per influx_connection.ts). OS/arch detection covers Linux amd64 + Darwin amd64 (via Rosetta on Apple-Silicon since upstream has no darwin-arm64 1.x release). Binary in bin-ext/influxdb/, data + logs + config in var-pryv/, idempotent, generates influxdb.conf with paths pinned. Rosetta-presence check on Apple Silicon catches Bad CPU type in executable with a clear install instruction. New companion storages/engines/influxdb/scripts/start (mirrors rqlite start: pidfile, background-default, foreground via DEVELOPMENT=true). Unblocks fresh-clone dev setup on macOS arm64 + on any non-Debian Linux.
storages/engines/postgresql/src/pg_connection.ts::parseInfluxSelect appends 'Z' to the captured time literal before new Date(), so JS interprets it as UTC (matching InfluxDB’s own semantics and series.ts:timestampToDateString’s intent).
Latent bug: series.ts:timestampToDateString emits InfluxQL literals like '1970-01-01 00:00:01.000000000' (no TZ marker — InfluxDB treats these as UTC). JS new Date() without TZ parses as LOCAL time. On a non-UTC dev machine (e.g. CEST/UTC+2) the literal became -3,599,000 ms instead of 1000 ms; the resulting nanos range matched zero rows in PG series_data, so any HFS read with a deltaTime offset returned []. Linux CI + Dokku production are UTC, so users were unaffected.
Verified: just test hfs-server 60/0 (was 59/1, [SDHF] [KC15] now passing); full PG matrix just test all 2312/0/8.
content.features (was content.extra) — features-negotiation contract driftOne-line fix in components/cmc/src/handleAccept.ts:94 paired with the @pryv/cmc@1.1.1 lib-js patch. The user-visible behaviour change (data-grant access now carries non-null clientData.cmc.features reflecting the negotiated offer features) is documented in CHANGELOG-v2.md. This entry covers the unit-test additions that pin the fix:
[HA01F] — handleAccept reads features from triggerEvent.content.features and stamps them on the data-grant; content.extra decoy is ignored.[HA01G] — when content.features is absent the data-grant’s clientData.cmc.features stays null even if content.extra is set (compat with legacy SDK callers).Wire-up + tests for the API-facing changes documented in CHANGELOG-v2.md (CMC security hardening). Adds:
components/cmc/src/hooks.ts: four new middleware factories — createAccessCreateForgePreventionHook, createAccessUpdateForgePreventionHook, createStreamDeleteReservedRootHook, createCounterpartyFromStampingHook, createEventsGetInternalGuardHook, createEventGetOneInternalGuardHook, createStreamsGetInternalGuardHook. All follow the pure-factory pattern (no api-server deps, unit-testable with fake deps).components/cmc/src/constants.ts: new isCmcInternalStreamId(streamId) predicate.components/cmc/src/errorIds.ts: new CLIENTDATA_CMC_FORBIDDEN (cmc-clientdata-cmc-forbidden) + reused CHAT_NO_REMOTE_APIENDPOINT semantics.components/api-server/src/methods/accesses.ts: wires forge-prevention hook into accesses.create + accesses.update.components/api-server/src/methods/events.ts: wires from-stamping + events.get/getOne internal guards.components/api-server/src/methods/streams.ts: wires streams.delete reserved-root + streams.get internal guard.Earlier Phase 2 phases (1.1, 2.1, 2.2, 3.1, 3.2) shipped on the same feat/cmc-phase-2-hds-readiness branch:
handleIncomingAccept looks up the capability access once, stamps inviteEventId on the inbox mirror from capabilityAccess.clientData.cmc.requestEventId. Closes cmc.revokeRelationship({inviteEventId}) doctor-side convenience path. lib-js: [CMCL1RC] + [CMCL1RD] cover the post-stamping lookup contract.capabilityMintHook reads event.content.request.expiresAt, bounds-checks to [60s, 30d], rejects out-of-range with cmc-capability-ttl-out-of-range. Default unchanged (7d when expiresAt omitted).handleChat / handleSystem consult clientData.cmc.features.{chat, systemMessaging} on the counterparty access at send time. Reject with cmc-chat-disabled / cmc-system-messaging-disabled when explicitly false (default-permit on omission).accessesUpdateHook runs accesses.update inside runWithSuppression so the post-hook doesn’t double-fire the peer notification.requestEventId real-flow) — new createCapabilityPostCreateHook middleware runs AFTER createEvent assigns the trigger’s real id, calls capability.setRequestEventIdOnAccess(...) to stamp it on the capability access’s clientData.cmc.requestEventId. Pre-fix the mint hook fired pre-persist when event.id === null so requestEventId was always null in production (HDS-reported).Test deltas (Plan 68 Phase 2 cumulative):
| Layer | Pre-Phase-2 | Post-Phase-2 | Delta |
|---|---|---|---|
| open-pryv.io cmc | 340 | 394 | +54 |
| open-pryv.io api-server | 1050 | 1055 | +5 (CN12/13/15/16/17) |
| lib-js pryv-cmc | 44 | 55 | +11 |
Plus 4 new deployed-infra validation scripts in _plans/68-cmc-datastore-atwork/tests/ (04-extended-messaging, 05-scope-update, 07-recapture, 08-sdk-handshake) — release-blocking gates before npm publish per Plan 68 Phase 6.
New docs/storage-isolation-for-parallel-tests.md enumerates every config key that a parallel-test fixture must override per mocha worker to avoid cross-worker collisions on shared PG databases, SQLite paths, ports, and rqlite endpoints. Audit confirmed every relevant key is reachable via config.set() and respected by its consumer — no code change needed; the doc is the canonical input for the per-worker test-helper that Plan 61 will ship.
Hardcoded fallbacks in bin/master.js (rqlite URL http://localhost:4001, raftPort 4002, dataDir var-pryv/rqlite-data) and components/api-server/src/server.ts (http:hfsPort default 4000) and components/messages/src/tcp_pubsub.ts (tcpBroker:port default 4222) are intentional for single-core production deploys; the per-worker fixture overrides them explicitly before ready() resolves. The doc pins the convention: code touching these keys must read through config.get() without an in-code literal fallback that would mask a missing config — let REQUIRED_WHEN catch it at boot.
Replace const x = config.get('slice') factory captures with lazy getters across components/api-server/src/methods/*.ts + the helpers that consumed captured slices (commonFunctions.getTrustedAppCheck, commonFunctions.catchForbiddenUpdate, eventsGetUtils.findEventsFromStore). After this commit, config.set() / injectTestConfig() / a future async config source reach every per-request callsite without a restart, and the PR-71-class bug shape (factory slice frozen at module init, missing a value populated later by override / plugin / extraConfig) is structurally impossible.
methods/account.ts: replace authSettings/servicesSettings captures with getAuth/getEmail getters. Drop the PR-71 request-time fallback (auth.passwordResetPageURL re-read + warn) — the §2A REQUIRED_WHEN check now guarantees the key is populated at boot. The RESET_LINK Pug substitution stays because it’s a real ergonomics improvement.methods/auth/login.ts: getAuth getter alongside the pre-existing getMfaConfig.methods/auth/register.ts: pass a getter function to new Registration().methods/events.ts: getAuth + getUpdates getters. The filesReadTokenSecret HMAC seed and updates.ignoreProtectedFields validator gate are the highest-severity audit rows — every attachment file-read token and every events.update validator depends on them.methods/streams.ts, methods/system.ts, methods/webhooks.ts: lazy getters for updates, services, webhooks respectively.methods/helpers/commonFunctions.ts::getTrustedAppCheck: signature changes from (authSettings) to (getAuthSettings). The closure-cached trustedApps list is dropped — re-parsed per-request from the fresh slice. Negligible cost; strictly more correct.methods/helpers/commonFunctions.ts::catchForbiddenUpdate: second arg ignoreProtectedFieldUpdates → ignoreOrGetter (back-compat: accepts literal OR function via typeof === 'function' branch).methods/helpers/eventsGetUtils.ts::findEventsFromStore: first arg filesReadTokenSecret → secretOrGetter (same back-compat shape).business/src/auth/registration.ts: constructor stores getServicesSettings (function); accepts literal OR getter. Welcome-mail send path reads this.getServicesSettings()?.email per-call.New ready() export on @pryv/boiler — the stronger-contract sibling of getConfig(). Documents the “config is ready to trust” contract at the call site: by the time it resolves, sync + async init has completed AND any registered boot-time validators (today: the config-validation plugin’s REQUIRED_WHEN + REPLACE-sentinel walk, which process.exit(1)s on problems) have run. Future Wave 2 work (PlatformDB-backed config, remote-file refresh) will extend the gate without touching every consumer.
ready() in components/boiler/src/index.ts. On the current codebase semantically equivalent to getConfig() — the value-add is the documented contract + the hook point for Wave 2.components/api-server/src/methods/*.ts (account, mfa, service, system, utility, auth/delete, auth/register, helpers/setCommonMeta, events, streams, trackingFunctions, webhooks, auth/login, helpers/updateAccessUsageStats) now use await ready() instead of await getConfig(). getConfig(), getConfigSync(), getConfigUnsafe() remain exported and unchanged for non-factory consumers.components/api-server/test/boiler-ready-seq.test.js — [CONFIG-RDY] describe block. Six unit tests pin the exported shape, the identity with getConfig() / getConfigSync(), the resolved test-config values, idempotency, and the key contract that Plan 70 §2C + Plan 61 both depend on: config.set() after ready() resolves is visible to the next .get() call.config/plugins/config-validation.js now refuses to boot when a feature-gated config key is missing or unset. Previously, a missing key silently degraded a downstream consumer at request time — the trigger for this work was PR #71, where auth.passwordResetPageURL could be absent at runtime when a deployment had the password-reset email feature enabled. The Pug template then rendered a broken href that some mail clients silently dropped.
REQUIRED_WHEN table in config/plugins/config-validation.js. Each entry pairs a colon-separated config path with a when(config) gating predicate. When the predicate is truthy, the key must resolve to a non-empty, non-sentinel value (REPLACE …, ${VAR} env placeholders, null/undefined, empty string all fail) — otherwise the existing process.exit(1) path fires after logging every problem in one pass.auth:passwordResetPageURL (when reset-password email is enabled — mirrors the runtime gating in methods/account.ts:174 against services.email.enabled), auth:adminAccessKey (always), auth:filesReadTokenSecret (always; the multi-core bootstrap bundle already enforces this — single-core deploys had no equivalent guard), letsEncrypt:atRestKey + letsEncrypt:email (when letsEncrypt:enabled === true).validate, checkRequiredWhen, isMissingOrSentinel, REQUIRED_WHEN. Lets the new [CV-REQ] unit-test suite exercise the validator with a fake config object without booting the boiler init lifecycle.components/api-server/test/config-validation-required-when-seq.test.js — [CV-REQ] describe block. Twelve unit tests cover the predicate matrix (enabled-missing / enabled-present / disabled), the isMissingOrSentinel classifier, and the shape of REQUIRED_WHEN. -seq because the api-server mocha hooks run a Platform DB integrity check around tests; the validator tests themselves do not touch storage.config/test-config.yml adds auth.passwordResetPageURL: http://test.pryv.local/reset-password so tests that override services.email.enabled = true (e.g. [G1VN], [HZCU] in account-seq.test.js) pass the new REQUIRED_WHEN check. The URL itself is not used by the test (the SMTP transport is mocked) but the key must be present to satisfy the boot validator.Side-note on the services.email.enabled config shape: today it’s an object ({ welcome: true, resetPassword: true }), inconsistent with the rest of v2’s flat boolean feature-gate convention. The REQUIRED_WHEN predicate for auth:passwordResetPageURL mirrors the existing runtime gating in methods/account.ts:174 to stay consistent during this change. Flattening the schema is a focused follow-up — see _plans/XXX-Backlog/SERVICES-EMAIL-FLATTEN.md in the macroPryv workspace.
The CMC docs (since Plan 68 first published IMPLEMENTERS-GUIDE.md in
commit 02b6d94) used a monitor.subscribe(streamId, callback) API
that doesn’t exist on @pryv/monitor. The actual Monitor API is:
const monitor = new pryv.Monitor(connection, { streams: [...] });
monitor.on('event', (event) => { ... });
await monitor.start();
Key semantic differences callers must understand (and the prior docs hid):
start() — to watch a different scope, construct another
Monitor (each shares the underlying socket.io connection via the
@pryv/socket.io add-on).'event' callback per Monitor. Branch on event.type /
event.streamIds[0] inside the callback.This commit sweeps all 14 occurrences in IMPLEMENTERS-GUIDE.md + 1 in README.md and replaces them with the correct pattern. The “bridge multi-tenant subscription” section gets a more accurate representation: one Monitor with a broad scope routes by streamId, OR two Monitors share the same underlying socket — either way it’s one WebSocket per bridge backend. Pure docs; zero behavior change; zero code change in CMC itself.
HANDOVER Q6 asked whether bridges managing thousands of patients need a new multi-tenant socket.io push channel (“inboxArrived”) to avoid opening N WebSocket connections. After working through it: the concern is a misread of CMC’s data direction. CMC traffic from a counterparty lands on YOUR streams, not on theirs:
:_cmc:apps:bridge-app:chats:<patient-slug>.:_cmc:inbox.So the bridge opens ONE socket.io connection on its OWN token, with
the SAME standard monitor.subscribe(':_cmc:inbox', ...) pattern
already documented, and receives push for every event from every
patient over that single connection. The counterparty slug in the
streamId identifies the patient. No new socket.io channel needed,
no new auth model.
The only N-connection concern is reading patient DATA streams (e.g. real-time vitals push per data-grant) — that’s a Pryv API surface question outside CMC’s scope.
Added a “Bridge / multi-tenant subscription” section to IMPLEMENTERS-GUIDE.md’s Socket.io reference making this explicit. Zero code change. Closes HANDOVER Q6.
HANDOVER Q5 asked whether the doctor’s “did patient X click my
invite yet?” UX needs a dedicated GET /cmc/capability/<id>/status
endpoint. The answer is no — after the Q1 Phase 1 lifecycle the
same data lives on the capability access (clientData.cmc.capability.state),
reachable via the existing accesses.get. Documented two query
paths in IMPLEMENTERS-GUIDE.md (“dashboard render” via
accesses.get + “real-time” via socket.io monitor on :_cmc:inbox).
Pinned the “no /cmc/* route namespace” rule as a fourth design
pillar in components/cmc/README.md (alongside “plugin, not storage
engine” + “zero new storage primitives”). Keeps the plugin a true
plugin — no API-surface ownership. Future CMC needs go via clientData
filters, trigger-event queries, or socket.io patterns, never via a
dedicated /cmc/* route.
event.createdByThe chat / system / scope-update / revoke handlers POST outbound to
the peer via the counterparty access. Without a structural guard, a
peer-delivered event would re-trigger dispatch on the receiving side
and POST right back — the classic A→B→A→B ping-pong. Previously the
rate-limiter (rateLimit.ts, 100/60s per (source, recipient)) was
the only thing cutting the loop, at the cost of a defensive ceiling
that doubles as both abuse defence and runaway-control.
This change splits the concerns:
components/cmc/src/dispatch.ts — new OUTBOUND_LOOPABLE_TYPES
set + isPeerDeliveredEvent helper. Before invoking a handler for
one of those types, the dispatch resolves event.createdBy to its
access on this mall; if the access has
clientData.cmc.role === 'counterparty' the event arrived from a
peer’s POST → return { status: 'skipped', reason: 'cmc-incoming-from-peer' },
no outbound. The rate-limiter remains as defence-in-depth for
abuse / quota.isOnInbox, and
the incoming variants do real protocol work.[CMCDISP-LOOP] test block — 9 new tests: 6 outbound types ×
skip-when-counterparty, plus non-counterparty-passes, missing-
createdBy-passes, lifecycle-type-exempt.cmc 327 → 336 (+9). CMCHS handshake 3/3 unchanged (the chat round-trip in CN13 now exits cleanly on the peer side instead of relying on the rate-limiter to cut the loop).
Per-app-code rate-limit override (the original HANDOVER Q4 ask) is captured as a separate backlog plan — operationally useful for high-volume collector apps, but no urgency now that loop defence sits at the right structural layer.
handleSystemScopeUpdate’s local-apply branch (the path that
synchronously updates the local data-grant before delivering the
peer notification) previously wrote newPermissions verbatim. If the
caller’s newPermissions omitted the CMC-machinery streams
(:_cmc:inbox create-only, the per-peer :_cmc:apps:*:chats:<slug>
and collectors:<slug> contribute permissions), those plugin-owned
permissions silently disappeared — and chat / system delivery from
this peer broke until the next handshake.
Auto-merge now reads the current access via mall.accesses.get,
identifies the existing :_cmc:* permissions as machinery, filters
the caller’s newPermissions to user-facing only, and overlays the
machinery back. Caller can include :_cmc:* perms — they’re
filtered out; the plugin owns those.
components/cmc/src/handleSystem.ts — local-apply branch updated.
~20 lines change.[HS28a] + [HS28b] tests cover both the auto-merge happy path
and the caller-supplied-CMC-perm filtering.Builds on the typed error-id catalogue: introduces a real two-state
lifecycle on the capability access (open → consumed /
invalidated) so re-clicks on an already-accepted single-use invite
are rejected at events.create time with a typed cmc-capability-consumed
error.id instead of silently re-running handleIncomingAccept (and
relying on the bug #12 duplicate-name fix to avoid a duplicate
back-channel mint).
consent/request-cmc.content.capability.mode
(optional, default 'single-use'): 'single-use' enforces the
state-flip, 'open-link' mints with mode set but state stays
'open' until Phase 2 lands (lifecycle enforcement for open-link
is the backlog plan).components/cmc/src/errorIds.ts — rename CAPABILITY_UNKNOWN
→ CAPABILITY_INVALID (BC: the prior name only existed on the
Plan 68 reopen branch, never on master). New constants
CAPABILITY_CONSUMED + CAPABILITY_INVALIDATED. cmc-capability-invalid
covers “never existed + expired past TTL”; cmc-capability-consumed
is the new state-flip rejection.components/cmc/src/capability.ts — mintCapability accepts
an optional mode param (defaults to reading
triggerEvent.content.capability.mode if present, else
'single-use') and stamps clientData.cmc.capability = { mode,
state: 'open', stateChangedAt } on the access. Two new exports:
findCapabilityAccess(userId, capabilityId) and
markCapabilityConsumed(userId, capabilityId). The pre-existing
legacy singleUse: true advisory flag is preserved.components/cmc/src/capabilityResponseHook.ts (new) —
events.create middleware that gates writes to
:_cmc:_internal:responses:<capId> by the capability access’s
state. 'consumed' → typed error cmc-capability-consumed;
'invalidated' → cmc-capability-invalidated. Legacy capabilities
(minted before this lifecycle field existed) and 'open' state
pass through.components/cmc/src/handleIncomingAccept.ts — after a
successful accept-arrives flow (back-channel access minted), calls
markCapabilityConsumed to flip state. Open-link mode capabilities
skip the flip (state stays 'open'). Best-effort; the back-channel
is already minted so the relationship is established even if the
state-flip fails (only the next re-click would mint a duplicate
back-channel — same as today’s behaviour).components/api-server/src/methods/events.ts — wires the new
hook into the events.create chain right after
cmcInboxWriteHook, before the persist step.cmc-capability-stale is
collapsed into cmc-capability-invalid; tombstone-based finer
discrimination is the explicit backlog work).[CMCCAP-LF] 7 new in capability.test.js covering
mint-time field stamps, find/markConsumed primitive, idempotency.
[CMCCRH] 6 new in capabilityResponseHook.test.js covering
passthrough + rejection paths. cmc total 312 → 325 (+13).
CMCHS handshake 3/0 unchanged.override-config.yml under NODE_ENV=testconfig/override-config.yml is .gitignored and intended only for
NODE_ENV=development node bin/master.js local iteration. When a developer
left it on disk, the boiler config loader (priority slot .1, above
everything else) merged it on top of test-config.yml for just test
runs as well, shifting service.api / auth.adminAccessKey / etc. out
from under tests that hardcode the canonical test expectations. Three
tests ([SVIF] config: serviceInfo, [RGRC] register-records-admin,
[SYRO] system route) plus the MFA-DELETE subroutes broke this way for
local development; CI never saw it because the file isn’t committed.
components/boiler/src/index.ts + src/config.ts — new
skipOverrideConfig option on boiler.init(). When true, the
override-config.yml load is skipped; the rest of the chain (memory →
test → argv → env → ${NODE_ENV}-config.yml → extras → default-config)
is unchanged.skipOverrideConfig defaults to true under
NODE_ENV === 'test' and to false otherwise. Production and
development runs continue to load override-config.yml. Callers
can still pass skipOverrideConfig: false to force-load.components/test-helpers/src/api-server-tests-config.ts passes the
flag explicitly for documentation; child processes spawned by
SpawnContext inherit NODE_ENV=test and therefore get the
default skip without each spawn-target having to know about it.Surfaces the stable kebab-case error.id strings the plugin emits via
content.failure.reason on failed trigger events as a single
authoritative catalogue. hds-macro Plan 59 Phase 5a’s per-outcome UX
can now pattern-match on these constants instead of parsing English
error.message.
components/cmc/src/errorIds.ts — CmcErrorIds constants
object (as const) enumerating 22 stable ids across capability
lifecycle, trigger content, handler routing, counterparty
resolution, access mint, outbound delivery, and chat/system
handler outcomes. Type-export CmcErrorId for TS consumers.readOfferViaCapability in
acceptOrchestration.ts now stamps CmcErrorIds.CAPABILITY_UNKNOWN
(cmc-capability-unknown) on HTTP 401 responses. Previously these
collapsed into the generic cmc-handler-offer-read-failed. Covers
three runtime cases that look identical from the client today —
token never existed, token expired (past TTL, plugin-GC’d), token
already consumed. Finer discrimination (cmc-capability-stale,
cmc-capability-already-accepted) requires capability tombstones
— design call deferred pending the 07-recapture probe outcome.cmc-handler-*
strings in handleAccept.ts stay as-is for back-compat with any
client matching on content.failure.reason. errorIds.ts exposes
them under semantic names but value-strings are unchanged.cmc index: cmc.CmcErrorIds + the namespace
cmc.errorIds (matches the existing cmc.constants / cmc.slug
pattern).[AO04B] asserts the 401 → cmc-capability-unknown mapping.
cmc 311 → 312 (+1).Follow-up to the Plan 68 TEST-GAP-DEBRIEF: Plan 68 shipped with
309 cmc unit tests + 1018 api-server tests but the real-deploy
validation suite still found 18 production-only code bugs +
3 fixture issues + 4 CI-only issues. The unit fakes accepted any
wire shape, and no test exercised the two-user handshake end-to-end.
This reopen closes the two highest-leverage gaps (debrief Phase 1 +
Phase 2). Phase 3 (deploy-smoke CI) is parked in the
_plans/XXX-Backlog/cmc-acceptance-harness/ backlog.
components/cmc/test/_fake-assertions.cjs exports
assertEventUpdateShape (rejects { id, update: {...} } — Plan 68
bug #1) and assertOutboundUrl (whitelists Pryv API paths +
permitted query params; rejects ?streamIds= — bug #2). Wired into
every fakeMall.events.update site (4 files) and every fakeFetch
site (7 files; outbound.test.js deliberately skipped since it
IS the URL builder under test).components/api-server/test/cmc-handshake.test.js exercises the
full request → accept → back-channel → chat handshake between two
real users on the in-process api-server. Three tests: [CN12]
happy-path handshake; [CN13] chat round-trip; [CN14] accept
re-delivery idempotency (regression for bugs #12 + #13). Transport:
a fetch shim that routes URLs whose host matches 127.0.0.1:3000
/ localhost:3000 (the test override-config’s service.api)
through coreRequest (the in-process supertest agent); pass-through
for any other host (data-types flat.json, rqlited at :4001).components/api-server/src/methods/events.ts —
the cmc plugin deps now capture globalThis.fetch lazily via
(url, init) => globalThis.fetch(url, init) instead of
fetch: globalThis.fetch, so a test that installs a fetch shim
after middleware registration is picked up by the dispatch loop.
Production: one extra closure indirection per call. Two call sites
tweaked (cmcDispatchMiddleware deps + the opt-in
startRetryLoopIfEnabled deps).just test cmc 309/0 unchanged. just test
api-server +3 from [CMCHS] (baseline preserved at 1018 → 1021
combined with the boiler skipOverrideConfig fix above).The :_cmc: namespace + write-hooks + orchestration handlers ship as a new top-level component components/cmc/. The plugin is loaded by the api-server like other components (event-content validation, capability-mint, inbox write-hook, dispatch middleware) plus a post-hook on accesses.update. No new storage engine — the entire plugin runs on standard per-user storage (PostgreSQL / MongoDB) + the existing pubsub layer.
components/cmc/ with module surface:
src/constants.ts — namespace + event-type constants; reserved-parent tree; classification predicates (isCmcStreamId, isAppNestedPluginStream, getAppCode); stream-id builders.src/slug.ts — counterpartySlug + parseCounterpartySlug (load-bearing -- separator).src/validators.ts — hand-rolled per-type content schemas for the 9 cmc/* event types.src/hooks.ts — content-validation hook + reserved-root rejection hook.src/inboxWriteHook.ts — :_cmc:inbox write-hook (role check + content.from stamping).src/capability.ts — single-use capability access mint + GC (creates real per-capability offer/responses streams + a shared access; cleaned up on consumption or TTL).src/capabilityMintHook.ts — fires on consent/request-cmc events.create.src/outbound.ts — federated HTTPS client (postToPeer + DeliverResult discriminated union; classify network/timeout/4xx/5xx).src/acceptOrchestration.ts — read offer via capability, build data-grant payload, deliver accept-via-capability.src/anchorStreams.ts — shared provisionAnchorStreams used by both sides at acceptance time.src/handleAccept.ts — accepter-side: creates data-grant + provisions anchors + delivers accept via capability + rollback on 4xx.src/handleRefuse.ts — accepter-side refuse delivery.src/handleIncomingAccept.ts — requester-side: mints back-channel access + provisions anchors when a consent/accept-cmc lands on the requester’s :_cmc:inbox.src/handleChat.ts, src/handleSystem.ts, src/handleRevoke.ts — per-type orchestration handlers.src/dispatch.ts — dispatch(...) type-router + createDispatchMiddleware (fire-and-forget). Auto-enqueues retryable failures.src/accessesUpdateHook.ts — createAccessesUpdatePostHook(deps) + runWithSuppression(fn) (AsyncLocalStorage-backed double-fire suppression).src/rateLimit.ts — per-worker sliding-window rate limiter (100 events / 60s window per (source, recipient) pair).src/retryQueue.ts — events-in-:_cmc:_internal:retries retry mechanism with exponential backoff + reason-based retryability classifier.src/retryScheduler.ts — RetryScheduler class wrapping the loop on an interval; operator-supplied userIdsProvider.:_cmc:* stream-ids route to LOCAL_STORE_ID passthrough (mirrors :_system:* precedent). Patched in components/mall/src/helpers/storeDataUtils.ts.components/api-server/src/methods/events.ts registers the three CMC write-hooks before createEvent + the dispatch middleware after notify. components/api-server/src/methods/accesses.ts registers cmcAccessesUpdatePostHook after emitUpdateNotifications. Both are fire-and-forget.events.ts and streams.ts skip slug normalization for ids starting with :_cmc: (otherwise colons would be munged and the path-style namespace destroyed).components/cmc/test/; api-server matrix at 1018 passing / 14 pending / 0 failing (PG default).account-seq.test.js [AC04 6041] (cumulative AC0X cycles + provisioning invalidate the test’s stored personal-access token). Workaround: lazy creation at first :_cmc:* write + at acceptance time (handleAccept + handleIncomingAccept use provisionAnchorStreams). Documented as TODO in business/src/users/repository.ts.A demo deploy on 2026-05-13 hit a ~20 min outage when new code shipped against an unmigrated schema: the operator’s override-config.yml carried migrations: { autoRunOnStart: false } and bin/master.js skipped the migration block in total silence — no log line at all. Every API call against the schema-dependent endpoints returned unexpected-error: column "head_id" does not exist.
The opt-out itself is intentional (operators want manual review on prod). The bug was the silent skip. master.js now always consults the runner.
storages/interfaces/migrations/applyOrAnnounce.ts — small policy helper. With autoRun=true it applies pending migrations (current behaviour byte-for-byte). With autoRun=false and pending migrations, it logs a top-line WARNING naming the count, the affected engine count, and a remediation hint (Run \node bin/migrate.js up` to apply.), followed by per-engine WARNING lines listing the pending filenames and current version. With autoRun=false` and nothing pending, a single info line confirms the runner was consulted.bin/master.js — replace the inline migration block with a single applyOrAnnounce call. Adds a warn(msg) closure mirroring the existing log(msg) so warnings hit both the boiler logger (logger.warn) and stdout (console.warn) with a [master] WARNING: prefix.components/api-server/test/migrations-skip-warn-seq.test.js — [MIGSKIP] describe block. Five pure unit tests ([MS01]–[MS05]) against fake runner + fake logger cover the run-applied / no-pending / skipped-no-pending / skipped-one-pending / skipped-many-pending-across-engines cases. Pin the exact info/warn line counts and message templates so the deploy-warning contract can’t regress silently.Behaviour matrix:
| autoRunOnStart | pending migrations | log level | shape |
|---|---|---|---|
| true (default) | any | info | unchanged from previous releases |
| false | none | info | 1 info line: Migrations skipped …; no pending … |
| false | ≥ 1 | warn | WARNING summary + per-engine WARNING line per row |
The [ACUP] test family validates Plan 66 end-to-end and uncovered several storage-path issues that needed fixing.
components/api-server/test/accesses-update.test.js — 17 tests across 7 sub-describes covering: composite-id bare-vs-versioned semantics, 409 stale-resource on update/delete, canUpdateAccess matrix (personal immutable, no self-update, app-can-update-managed-shared), Rule A/B/D (shared scope ⊆ managing app, narrowing parent rejects with offendingChildren, expiry chain on update + create), soft-deleted → unknownResource, accesses.getOne with current head / obsolete composite / unknown id / ?includeHistory=true, pubsub coarse + fine-grained notifications, and checkApp head-only semantics.storages/engines/postgresql/src/user/BaseStoragePG.ts:DEFAULT_COLUMN_MAP — added createdBySerial → created_by_serial and modifiedBySerial → modified_by_serial. Without these, PG lowercased the unmapped camelCase identifier and raised column "modifiedbyserial" does not exist on the first accesses.update.snapshotHead rewrites on both engines — route the history-row insert through the standard insertOne path so applyDefaults’s integrity-recompute fires against the snapshot row’s actual fields. The original approach copied the head’s integrity hash verbatim, which never matched the snapshot row’s (fresh _id + new headId) shape and the periodic integrity-final-check rejected every snapshot. Side effect: snapshotHead is now callback-based to fit BaseStorage’s existing callback API.headId strip (PG AccessesPG.rowToItem delete item.headId + Mongo stripHeadId converter). The integrity hash was including headId at insert and the strip at re-read made every recompute miss. The strip moved to composeWireAccess (api-server seam) so the hash is consistent inside storage AND headId still never appears on the wire.composeWireAccess now also strips headId, alongside the internal serial fields.components/test-helpers/src/dependencies.ts:init runs the Plan 32 migration runner so the test DB matches the deployed shape. bin/master.js calls it in production; the test harness used to skip it, leaving the unique-token index without the new head_id IS NULL predicate and causing Plan 66’s first accesses.update to hit a duplicate-token violation.components/api-server/test/migrations-runner-seq.test.js:beforeEach resets each engine’s schema_migrations tracking state before every test. The Phase F migration-runner invocation in dependencies.init would otherwise leave the tracker at v1 and trip [MR01]’s “fresh engines report v0” assertion.audit/src/Audit.ts:buildDefaultEvent — reads context.access.serial (set by AccessLogic’s deepMerge(this, access) from the storage row). When non-null, the event’s streamIds array now carries both access-<base> and access-<base>:<serial> followed by the existing action-<methodId>. When null, behaviour is identical to before (single access-<base> entry). Cost: ~30 bytes per audit row on versioned-access activity. No schema change.api-server/src/socket-io/Manager.ts:messageFromPubSub — handles both legacy string payloads (event-name only, used by eventsChanged / accessesChanged / streamsChanged) and the new structured object payloads ({ type, …data }). For structured payloads, looks up the socket event name by payload.type and forwards the entire payload as the socket.io message argument. Backwards-compatible: existing listeners that subscribe to 'accessesChanged' keep receiving arg-less calls.messageMap[pubsub.ACCESS_UPDATED] = 'accessUpdated' added — without this entry the structured payload would log XXXXXXX Unknown structured payload and silently drop.methods/accesses.ts:emitUpdateNotifications — restructures the second emission so the payload is the entire structured object ({ type, accessId, serial }) rather than a 3rd arg to pubsub.emit (which only takes (eventName, payload) and would silently drop the data). The first emission stays as a string USERNAME_BASED_ACCESSES_CHANGED for the legacy accessesChanged socket event.Internal plumbing for the new accesses.getOne and the composite-id wire format (see CHANGELOG-v2.md).
composeWireAccess(row, historyOfBase?) in business/src/accesses/refs.ts. Takes a storage row and emits the wire-format access object: composes the composite id / createdBy / modifiedBy from the row’s bare <col> + sibling <col>Serial columns, and strips the now-redundant serial / createdBySerial / modifiedBySerial fields so the response stays inside the schema’s additionalProperties: false whitelist. When historyOfBase is passed (history-row case), the wire id uses that base instead of the storage’s fresh history-row id — so <base>:<serial> always means “this version of base.”composeStoredRef(storedRef, serial) helper inside refs.ts. Handles the <base> <callerId> tracking-author format: splices the :<serial> into the access-id slice and preserves the space-separated caller tail. (MethodContext.ts already parses callerId from the first space, so this round-trips cleanly.)Accesses.findHistory(userOrUserId, baseId) on both engines. Returns history rows where headId === baseId, sorted by modified ASC (oldest first). PG uses WHERE user_id = $1 AND head_id = $2; Mongo uses { userId, headId: baseId }.sort({ modified: 1 }). Each engine’s existing rowToItem / applyItemFromDB pipeline strips headId before returning — the caller (composeWireAccess) gets the base from the query parameter.accesses.getOne method handler findOneAccess in methods/accesses.ts. Parses composite id, looks up the head by base, applies visibility check (app callers see only self + their managed shareds, by base), then either (a) returns the head when bare/serial matches, (b) returns the historical snapshot + current hint when serial < head’s, or (c) unknownResource for never-existed serials. ?includeHistory=true appends the full chronological history.methodsSchema.getOne entry — params: { id, includeHistory? }, result: { access, current?, history?: [...] }.GET /accesses/:id → accesses.getOne, with tryCoerceStringValues on includeHistory so the boolean comes through correctly from query string.AccessLogic.can('accesses.getOne') — added to the switch, returns !isShared() (same gate as accesses.get).accesses.getOne registered in audit ALL_METHODS (audit/src/ApiMethods.ts) — without it, API.register throws at boot and every api-server test that initializes the API crashes in the before all hook. (First Phase D test run had 298 failures from this single oversight.)composeWireAccess applied to: accesses.get (list), accesses.get deletions, accesses.create result, accesses.update result (replacing the ad-hoc id rewrite from Phase C), accesses.checkApp (matching + mismatching). Audit-driven and storage-driven internals continue to operate on the raw (base, serial) columns — composition is purely a presentation-layer concern.snapshotAndApplyUpdate rewritten to delegate id composition to composeWireAccess instead of building the composite string in-place. Behaviour identical; less duplication.Wire-up for the revived accesses.update (see CHANGELOG-v2.md). Internal-only plumbing notes:
Accesses.snapshotHead(userOrUserId, baseId) on both engines. Reads the current live head row by base, clones every column/field, replaces id/_id with a freshly-minted cuid and sets head_id/headId to the original base. The unique-token partial filters (WHERE deleted IS NULL AND head_id IS NULL in PG; partialFilterExpression: { deleted: $type null, headId: $type null } in Mongo) exclude the new history row, so the head and snapshot can share a token without violating uniqueness.api-server/src/methods/accesses.ts): basicAccessAuthorizationCheck → schema-validate → loadAccessForUpdate → enforceUpdateChainRules → snapshotAndApplyUpdate → emitUpdateNotifications. loadAccessForUpdate parses the composite id, conflict-checks serial, treats soft-deleted as unknownResource, and gates on AccessLogic.canUpdateAccess. enforceUpdateChainRules resolves expireAfter → expires and applies Rules A/D for shared targets + Rules B/C/D for app targets (iterating the user’s live shareds, matching by base).AccessLogic.can('accesses.update') — added to the switch, returns !isShared(). Without this, basicAccessAuthorizationCheck threw on the unknown methodId and the handler crashed with unexpected-error.update.modifiedBy is set by the standard MethodContext.updateTrackingProperties (caller’s bare base). update.modifiedBySerial is set explicitly from context.access.serial (null today since AccessLogic doesn’t yet carry serial — Phase D will plumb it). update.serial is set to (prev || 0) + 1.stale-resource (ErrorIds.StaleResource, 409) added to errors/{ErrorIds.ts,factory.ts}. Used by accesses.update and accesses.delete on composite-id mismatch.accesses.delete composite-id check — checkAccessForDeletion parses the composite, conflict-checks the serial against the head, then rewrites params.id to the bare base for downstream stages (findRelatedAccesses, deleteAccesses) which all expect bare ids.accessesMethods.ts gains __ex_update = { params: { id, update: access(Action.UPDATE) }, result: { access: access(Action.READ) } }. The UPDATE-action access schema already exists (Action.UPDATE branch) with the mutable-fields whitelist.[11UZ] app-cannot-update-sibling-app → 403, [U04A] unknown-id-on-PUT → 404, [1WXJ] create-only-shared-cannot-update → 403, [OS36] deleted in update body → invalid-parameters-format. Test IDs preserved; assertions and descriptions updated in place.Internal-only utilities that the upcoming accesses.update (Phase C) will consume. No behavior change today besides the Rule D retrofit on accesses.create (see CHANGELOG-v2.md).
components/business/src/accesses/refs.ts — parseAccessRef(ref) and serializeAccessRef({ base, serial }). Wire format is bare cuid when no serial, <base>:<serial> otherwise; separator is : (URL-safe, never appears inside a cuid/cuid2 id). Throws on malformed input.ACCESS_UPDATED = 'access-updated' (components/messages/src/constants.ts + index.ts). Payload shape (set when Phase C fires it): { accessId: '<base>:<serial>', serial: number }. Companion to the existing USERNAME_BASED_ACCESSES_CHANGED event so fine-grained subscribers can act on a specific update without refetching.AccessLogic.canUpdateAccess(target) — encodes the §3 caller-vs-target matrix from the plan: no self-update (parses both ids so a future composite ref still matches the caller’s bare base), personal-immutable, personal-can-update-non-personal, app-can-update-only-shared-it-manages (chain match by base via parseAccessRef(target.createdBy).base === this.id), shared-cannot-update-anything. canUpdateAccess is the gate; chain-rule application (A/B/C/D on changes) is enforced by the call path in Phase C.Lays the schema and storage-layer plumbing for the upcoming accesses.update revival. No API surface change yet (the wire-format composite id <base>:<serial> and the revived method land in later phases). Both baseStorage engines (PostgreSQL, MongoDB) get the same treatment; engines that don’t store accesses (sqlite, rqlite, filesystem, influxdb) are untouched.
accesses table: serial INTEGER, head_id TEXT, created_by_serial INTEGER, modified_by_serial INTEGER — all nullable. Added via the new migration storages/engines/postgresql/migrations/20260512_132200_access_versioning.js (Plan 32 framework). Same migration tightens the two unique indexes (idx_access_token, idx_access_name_type_deviceName) to predicate AND head_id IS NULL so future history rows don’t collide on token/(name+type+deviceName) uniqueness, and adds a new partial index idx_access_head_id ON accesses(user_id, head_id) WHERE head_id IS NOT NULL for history-lookup queries. SCHEMA_SQL in DatabasePG.ts keeps the pre-Plan-66 index predicates because CREATE TABLE IF NOT EXISTS is a no-op on existing installs — fresh installs converge once the migration runs at boot.storages/engines/postgresql/migrations/package.json with { "type": "commonjs" } so migration files keep the README’s module.exports = { async up () {} } shape despite the engine package being "type": "module". Without this scope-override the runner’s require() failed with module is not defined in ES module scope.AccessesPG — hasHeadIdCol = true. rowToItem strips headId from the returned item (internal storage marker, never on the wire). BaseStoragePG.findOne now adds query.headId = null when hasHeadIdCol is true so the auth-by-token path and findOne({ id }) can never return a history row.Accesses — both unique indexes’ partialFilterExpression extended with headId: { $type: 'null' }; new non-unique { headId: 1 } partial index for history queries. New setHeadIdNullIfMissing converter forces live-row inserts to set headId: null explicitly (the $type: 'null' partial filter only matches BSON null, not missing fields). New stripHeadId converter on itemFromDB keeps headId off the wire. New bootstrap() method drops the pre-Plan-66 unique indexes (token_1, name_1_type_1_deviceName_1) and backfills headId: null on legacy rows so they re-enter the new unique-token set; idempotent over fresh DBs (NamespaceNotFound/IndexNotFound are silent successes).BaseStorage.findOne — adds query.headId = null. Equality-null in Mongo matches both missing-and-null fields, so this is a no-op for tables without a headId field.initStorageLayer is now async — Mongo’s flavor awaits storageLayer.accesses.bootstrap() after construction; PG’s stays sync (returns undefined). StorageLayer.init adds await accordingly.accesses.update and the snapshot logic); no composite-id format on the wire yet (Phase D); accesses.update still returns goneResource. The schema and storage just sit ready.bootstrap test coverage; full Mongo re-run gated on Phase C when behavior actually diverges.serial and headId to the canonical-fields list and rolling the integrity-format version. In practice no bump was needed: @pryv/stable-object-representation/access.js:stringifyAccess0 already serialises every field on the access object (deep-clone + strip known volatile fields like integrity / lastUsed / calls / apiEndpoint). New nullable fields serial / head_id simply roll into the hash automatically — never-updated accesses have absent values (no hash change), versioned accesses include serial: <N>, and history rows include both serial (frozen) and headId (base). The ACCESS:0: prefix stays as-is. If we ever want a tamper-detection guarantee that REJECTS a serial removal (vs just detecting an integrity mismatch), bumping to ACCESS:1: becomes warranted then; current behavior is detection-via-mismatch.storages.series.engine flipped from influxdb to postgresqlconfig/default-config.yml: storages.series.engine: influxdb → postgresql. Operators who want influxdb-backed series storage must now set storages.series.engine: influxdb explicitly in override-config.yml (and run a reachable influxd on http.ip:storages.engines.influxdb.port).config/test-config.yml: pins series.engine: influxdb explicitly. Test matrix behavior preserved — series tests still run against influxdb. Pin can be removed once the matrix is re-validated against postgresql series.cluster.hfsWorkers > 0 with the inherited influxdb default produced a silent footgun: HFS workers came up, requests reached the worker, but every write hit a missing backend. PostgreSQL seriesStorage has been first-class since Plan 19. Flipping the default removes the trap for fresh installs.engine: influxdb set explicitly are unaffected. Deploys without an override that have an actual influxd running need to either keep it running (and add the explicit override) or migrate any existing series data — note: PG and influxdb series stores are NOT cross-compatible, this is a forward-going default.components/api-server/src/hfsIngress.ts — buildHfsIngress({ hfsHost, hfsPort, logger }) factory returns a (req, res, fallback) => void dispatcher.components/api-server/src/server.ts now constructs https.createServer(opts, requestHandler) where requestHandler is a wrapper that calls the HFS dispatcher first, then falls through to app.expressApp. The fall-through preserves the prior behavior for all non-HFS paths.^/<user>/events/<id>/series and ^/<user>/series/batch are forwarded via http.request to http.ip:http.hfsPort (default 127.0.0.1:4000), streaming both request and response bodies. JSON-shaped 502 on upstream unreachable.[HFSI] cases in components/api-server/test/hfs-ingress.test.js cover regex matching, dispatch, fallback, and 502.components/ingress/ component — filed at _plans/XXX-Backlog/EXTRACT-INGRESS-COMPONENT.md.getConfigSync() companion; defer module-top reads@pryv/boiler exports getConfigSync() — sync access to the fully-loaded config. Throws if init() hasn’t been called or if async config-loading is still pending. Use anywhere a sync read is needed at request/test time post-init.getConfigUnsafe(warnOnly) retained as the documented escape hatch for genuine pre-init reads (returns partial config; with warnOnly: true warns instead of throws). Two production sites remain on it after the cleanup: components/business/src/integrity/integrity.ts:9 (module-top capture; preserves cross-process symmetry between mocha-parent and api-server forked-child that the fixture-time hash compute relies on) and components/storage/src/index.ts:_ensureMongoDatabase (test-helpers/dependencies lazy-loads MongoDB at module-load). Plus 3 test-helpers fixture files (data.ts, dynData.ts, dependencies.ts) which run pre-init by lifecycle.getConfigUnsafe(true) reads into function bodies:
components/cache/src/index.ts — loadConfiguration() is now async and awaits getConfig(). Module-bottom auto-call becomes fire-and-forget with stderr log on misconfig; cache stays inactive on failure instead of killing the worker. Cache ops short-circuit on !isActive so the brief async window matches legacy partial-config behavior.components/previews-server/src/attachmentManagement.ts — module-top previewsDirPath capture → lazy-memoized getPreviewsDirPath(). All 3 callers run at request/test time.components/previews-server/src/runCacheCleanup.ts — module-top sync config reads → async IIFE that awaits getConfig() before constructing Cache. Strictly safer than the legacy race against partial config.components/api-server/src/routes/register.ts — drops (true) warnOnly; caller is express bootstrap post-await getConfig().components/test-helpers/src/data/events.ts — module-top Array.map calling integrity.events.set(event, false) synchronously at module-load → idempotent ensureIntegrity() helper. Called from resetEvents() and from helpers-base.ts beforeAll. Decouples fixture integrity from module-load timing.getConfigUnsafe() (no warnOnly) → getConfigSync() at 7 production sites: api-server/{API,middleware/errors,routes/auth/login,routes/register}.ts, business/src/accesses/AccessLogic.ts, previews-server/src/attachmentManagement.ts, utils/src/api-endpoint.ts.1857/1 (only pre-existing [ASTE] flake), Mongo 1849/2 (pre-existing [ASTE] + [3TMH] timing-sensitive webhooks test that passes on standalone re-run).package.json for components + storages + engines now has "type": "module"; every .ts source file uses import { createRequire } from 'node:module'; const require = createRequire(import.meta.url); to keep its existing CJS-style internal require() calls working under the ESM module loader. Default exports use export default X; named exports use export { X } or export const X = .... Sub-package.json {"type":"commonjs"} overrides drop bin/server (each component’s CJS fork target), three test/helpers/ directories that use the module.exports = { ...require('test-helpers') } spread-mutation pattern, and a few CJS-only fixture/test-engine subtrees..mocharc.cjs files stay CJS as Node requires (mocha config is loaded via CJS); the _ts-register shim is loaded via require: from the base mocha config.components/middleware/src/project_version.ts — added an ESM-safe fallback for readStaticVersion() that walks upward from the file’s own __dirname looking for .api-version. Fixes a regression that surfaced once forked children (HFS test-helpers, accessStateWorker, etc.) became ESM: process.mainModule and require.main both went undefined, the function returned null, and the api-version header fell through to the git-describe stamp — breaking every consumer that asserted /^\d+\.\d+\.\d+/ on it.components/test-helpers/src/helpers-base.ts:121-128 — the dynamic for (const method of methods) { const loaded = require('api-server/src/methods/' + method); if (typeof loaded === 'function') { ... } } loop was silently skipping every method registration once api-server became ESM (Node 24 require(esm) returns the namespace object, not the function). Fixed: unwrap loaded.default first. This single fix turned a “119 failing” intermediate api-server run into 978/0.storages/index.ts) exposes database, databasePG, connection, storageLayer, userAccountStorage, usersLocalIndex, platformDB, auditStorage, seriesConnection, dataStoreModule as live-bound let exports updated by _refreshExports() after init() and reset(). New test-only _setPlatformDBForTest(db) setter replaces the now-incompatible Object.defineProperty(storages, 'platformDB', ...) pattern (ESM module namespace properties are non-configurable).system-streams/index.ts — top-level let bindings replace the previous module.exports = X; X.foo = Y mutation pattern; live-bound exports propagate state changes (e.g. accountChildren reassignment in initializeState()) to all consumers without breaking the namespace.storages/engines/postgresql/test/{global,audit-conformance}.test.js had a pre-existing _internals.getLogger = X direct assignment that was always wrong (the export defines getLogger as a get-only property; CJS plain-object assignment silently shadowed the getter, ESM frozen namespace throws Cannot add property getLogger). Fixed by destructuring { _internals } and using the proper _internals.set('getLogger', X) API.module.exports = ClassName default-exports in test fixtures (HttpServer in business + api-server, SyslogWatch, InfluxConnection.test.js’s conformanceTests factory) became export default ClassName, with consumers patched to .default-unwrap.bin/_ts-register.js is retained — its require.extensions['.ts'] = require.extensions['.js'] mutation is still load-bearing for the three CJS helper subdirectories and the forked CJS bin/server targets that do extensionless deep .ts requires (e.g. require('test-helpers/src/helpers-base'), require('messages/src/cluster_kv')). Phase 5g experiment (commenting out the line) regressed the matrix; full removal moves to backlog.1857/3, Mongo 1852/1 — failures are pre-existing flakes ([ASTE] audit matrix-state, [3TMH] webhook timeout, occasional webhook DELETE app-token race with stale DB residue). No ESM regressions; both engines are correctness-equivalent to the pre-5f baselines (1858/2 and 1853/0).await is now syntactically available in any .ts file. The storages barrel still uses async init() (consumer-compatible with Node 24 require(esm) interop); switching it to TLA is the unblocker for XX-finalize-storages-plugin-later and is left as a follow-up..js → .ts for the 10 production source files in storages/ that Phases 1–3 left untouched (they were outside the storages/interfaces/* and storages/engines/* scope): storages/{index,internals,manifest-schema,pluginLoader}.ts, storages/shared/{DeletionModesFields,treeUtils,localStoreEventQueries}.ts, storages/datastores/account/{index,AccountUserEvents,AccountUserStreams}.ts. Added module markers + two minor TS-narrowing fixes: optional positional args on fieldToEvent (fieldName, value, streamConfig, time?, createdBy?) and const engines: Record<string, any> = {} in pluginLoader.components/ + storages/ production source is now 100% TypeScript. Only .mocharc.js test-config files in storages/engines/* remain .js (intentional — mocha config). Runtime is still source-loaded via bin/_ts-register.js shim. Sets up Phase 5c.2 (rewrite source require() → import).module/moduleResolution: nodenext (Plan 57 Phase 5b)tsconfig.json — module: commonjs → nodenext, moduleResolution: node → nodenext. Under nodenext, tsc decides per-file CJS-vs-ESM emit based on the closest enclosing package.json "type" field. Since no package.json declares "type": "module" yet, all files still emit CJS — no runtime change. Validates the toolchain switch in isolation; per-file emit format flips in Phase 5c onwards as packages opt into "type": "module".dist/components/api-server/src/server.js still has Object.defineProperty(exports, "__esModule") + require() (CJS shape). PG matrix 1860/0 unchanged.storages/test/barrel-init-order.test.js — 5 [BIO] cases pinning the current CJS barrel contract: pre-init getter access returns undefined (does not throw), pluginLoader is exposed regardless, init() is idempotent, reset() returns to pre-init state. ESM with top-level await in the barrel would fundamentally change pre-init semantics — without this pin a regression direction (silent vs throw) lands undetected on feat/ts-esm.components/messages/test/worker-fork-ts-loading.test.js + fixtures/wftl-worker.js — 2 [WFTL] cases pinning the Phase 1 NODE_OPTIONS-shim mechanism: parent process has NODE_OPTIONS=--require=…/_ts-register, forked children inherit it and can require() a .ts source file without explicit shim load. Phase 5e drops the shim — must remain functionally equivalent under native ESM .ts loading or every child_process.fork() target (cluster_kv master, accessStateWorker, hfs background) breaks.storages/test/engine-runtime-contract.test.js — 33 [ERC] cases, one per (engine × required export) combination across all 6 engines (filesystem, mongodb, postgresql, sqlite, rqlite, influxdb). Asserts each engine’s loaded module exports the methods that pluginLoader.REQUIRED_EXPORTS demands for each storageType in its manifest. ESM export { foo } vs CJS module.exports = { foo } typically shifts exports under a default namespace — without this pin the silent shape change crashes consumers at deferred runtime, not at compile time..js → .ts for every source file under components/api-server/src/ across 8 sub-folders: top-level (5: API, Result, application, expressApp, index, server), methods/ (12), methods/auth/ (3), methods/helpers/ (6), methods/streams/ (5), middleware/ (3), routes/ (15 incl. routes/auth/ + routes/reg/), schema/ (22), socket-io/ (2). Total: 82 source renames. Plus the established Phase 4 fix-up patterns: module markers, : any casts on mutable response/options/query objects (e.g. routes/reg/access.ts response, methods/accesses.ts query, routes/events.ts params, schema/{stream,user,access}.ts mutable schema base), class field declarations (Server.isAuditActive), Promisecomponents/api-server/package.json main bumped src/index.js → src/index.ts.application.ts:259 — pre-existing typo this.customAuthStep (undefined) at logger.debug 2nd arg, masked the latent crash where passing a function to boiler’s inspectAndHide throws JSON.parse('undefined') (JSON.stringify(fn) === undefined). Removing the 2nd arg entirely (the boolean is already in the message string) preserves the log information without crashing the worker on any test that exercises getCustomAuthFunction().methods/accesses.ts:362-368 — pre-existing operator-precedence bug since 685034dd (2023-10-13): if (!accessToDelete.type === 'personal') always evaluates to false (precedence: ! before ===), so the early-return branch is dead and findRelatedAccesses runs for every access type. Preserved current observed behavior with a defensive cast and a marker comment; tracked at _plans/XXX-Backlog/ACCESSES-FIND-RELATED-PRECEDENCE-BUG.md.bin/_ts-register.js shim from Phase 1; deployability invariant preserved. All components/ + storages/ source code is now 100% TypeScript — Phase 5 (ESM flip + drop the shim) is unblocked..js → .ts for every source file under components/business/src/ across 14 sub-folders: top-level (2), accesses/ (2), acme/ (12), auth/ (2), backup/ (3), bootstrap/ (10), integrity/ (4), mfa/ (7), observability/ (6), series/ (8), system-streams/ (1), types/ (5), users/ (5), webhooks/ (2). Total: 67 source renames. Module markers added (import type {} from 'node:fs'), class field declarations (foo: any;) added wherever this.foo = ... was set in constructor, static class members declared (static PERMISSION_LEVEL_* on AccessLogic, static replaceAll/replaceRecursively on mfa/Service), destructure-default opts annotated { x, y }: any = {}, mutation-after-new Error(...) patterns annotated const err: any = new Error(...), Object.entries(map) widened with as Array<[string, any]> casts at iteration sites, Object.values(map) casts to as any[], Promise<void> typed on void-resolving constructors, optional positional args ?: on _restoreSingleUser/_verifyEventIntegrity/setUserPassword/#issue. One existing-bug callout: IntegrityStream constructor cast as any because the upstream type signature lacks the algorithm arg.components/business/package.json main bumped src/index.js → src/index.ts so the workspace symlink resolution + bin/_ts-register.js shim resolve .ts first.bin/_ts-register.js shim from Phase 1; deployability invariant preserved..js → .ts for every source file under components/middleware/src/ (15), components/audit/src/ (16), components/hfs-server/src/ (16), and components/test-helpers/src/ (26 — 21 top-level + 5 in data/). Total: 74 source renames plus components/api-server/.mocharc.js require: 'test-helpers/src/helpers-c.js' → '…/helpers-c.ts'. Module markers added (import type {} from 'node:fs') on every renamed file (script-vs-module disambiguation), default-arg objects annotated : any on consumers that mutate them, Promise<void> typed on void-resolving constructors, optional callback params (?:) on tail-call recursive helpers, super_ accesses on util.inherits classes cast (SubClass as any).super_, variadic arguments rewritten as ...rest: any[], and one errorlogger → errorLogger typo fix on Server (lowercase declaration was inconsistent with assignment + use sites).package.json main bumped src/index.js → src/index.ts (or src/server.js → src/server.ts for hfs-server) on each touched component so the workspace symlink resolution + bin/_ts-register.js shim resolve .ts first.bin/_ts-register.js shim from Phase 1; deployability invariant preserved.package.json — typescript ^5.9.3 and @types/node ^24.0.0 added to devDependencies. TypeScript was previously pulled transitively via neostandard; now it’s pinned directly so the build pipeline doesn’t break when neostandard updates.tsconfig.json — reshape from a JSDoc-only checker config to an emit-capable config: target: es2022, module: commonjs, outDir: ./dist, rootDir: ., esModuleInterop: true, skipLibCheck: true. checkJs flipped to false — the previous checkJs: true baseline accumulated 7,200+ silent errors that nobody enforced; quality going forward is gated through new .ts conversions instead. allowJs: true remains so the codebase keeps building during incremental conversion. Includes broadened to cover components/, storages/, and bin/ (was components/ + test/ only).justfile recipes — just typecheck (tsc –noEmit) and just build (tsc emit to ./dist)..gitignore — /dist ignored.bin/master.js still loads from source under components/ + storages/; dist/ is informational until later phases convert sources to .ts and flip the runtime entry. The deployability invariant — every commit on this branch keeps bin/master.js runnable from source — is preserved.$inc JSONB-path collision — multiple assignments to same columnstorages/engines/postgresql/src/user/BaseStoragePG.ts — _buildUpdateClauses $inc loop emitted one SET col = jsonb_set(...) clause per dotted-path entry. When a single update contained ≥2 entries sharing the same top-level column (e.g. $inc: { 'calls.events:get': 1, 'calls.accesses:get': 1 }, produced by every batched API call when accessTracking.isActive: true), Postgres rejected the UPDATE with multiple assignments to same column "calls". Per-method counters were silently dropped (the storage error was caught + logged by updateAccessUsageStats.js after the API response had already returned 200). Fix groups dotted-path $inc entries by top-key and emits ONE nested jsonb_set(jsonb_set(..., k1, v1), k2, v2) clause per column. Each nested call reads the original column value (col->>$key), preserving Mongo $inc disjoint-paths semantics.[BTRK] regression in components/api-server/test/root-seq.test.js: issues a real batched POST /<username> with events.get + accesses.get and asserts both per-method counters move (and by the same delta). Pre-fix the assertion fails because the SQL crashes and counters stay at baseline.components/audit/src/syslog/Syslog.js — the winston-syslog transport’s underlying unix-dgram socket emits 'error' on the first send when the configured socket path doesn’t exist (typical containerized deploy with no /dev/log). winston-transport extends stream.Writable, and Writable.emit('error', err) with no listener throws synchronously → worker exits code 7 → cluster master recycles → user-visible: registration row landed in users_index but the auth poll on core-<id>.<domain>/reg/access/<key> times out with no token issued. Now transport.on('error', err => logger.warn('audit syslog dropped', err)) so audit emits become best-effort observability instead of a load-bearing path.config/default-config.yml — audit.syslog.active: false (operator-facing, in CHANGELOG-v2.md). Defense in depth: even if the listener regresses, the gate at audit/src/syslog/index.js:18 still short-circuits the whole code path.components/business/src/bootstrap/Bundle.js: BUNDLE_VERSION
bumped 1 → 2. v2 adds an optional
platformSecrets.letsEncrypt.atRestKey field carrying the cluster-wide
AES-GCM key used by AtRestEncryption.Bundle.validate() accepts any version in 1..BUNDLE_VERSION
(was strict equality on the current version). Producers always emit the
latest version; consumers reject only forward-compat unknown versions or
version <= 0. Restores graceful upgrade across mixed-version clusters
during the rolling-out window.Bundle.assemble() only emits platformSecrets.letsEncrypt
when the input supplies an atRestKey — keeps v2 bundles minimal when
the issuing core has no LE secret to ship. Bundle version stays 2
regardless.applyBundle.writeOverrideConfig(): when the bundle ships
platformSecrets.letsEncrypt.atRestKey, write it under
letsEncrypt.atRestKey in the joiner’s override-config.yml.cliOps.newCore() accepts secrets.letsEncryptAtRestKey;
forwards to Bundle.assemble. bin/bootstrap.js reads
config.get('letsEncrypt:atRestKey') and threads it through (only when
it’s a real value — REPLACE ME placeholder is filtered out by
isUsableSecret).[BUNDLE] +6 cases (omits/carries letsEncrypt, accepts v1
shape, accepts v2 shape, rejects version 0, rejects malformed
letsEncrypt); [APPLYBUNDLE] +1 case (writes
override.letsEncrypt.atRestKey); [BOOTSTRAPE2E] +1 case (issuer →
consumer round-trip of atRestKey). just test business: 363/0 (was
354).cluster_kv primitiveA class of bugs where module-scope new Map() looks fine in single-process
tests but breaks under cluster.fork() because each worker holds its own
copy. Surfaced in production as a 50 % auth-poll failure rate
(/reg/access/:key polls round-robin across workers; the second poll lands
on a worker whose Map is empty).
components/api-server/src/routes/reg/accessState.js — replaces
the in-memory new Map() with PlatformDB-backed storage (rqlite
keyValue rows under access-state/<key>). API turned async; the route
in routes/reg/access.js is now async with try/catch wrappers. POST
splits into buildState() → URL decoration (pollUrl/authUrl) →
persist() so the URLs computed from per-core routing land in the
stored state without an extra round-trip. Cluster-wide AND
restart-survivable for free; the lazy expire on get matches the
existing tls-cert/* posture.storages/interfaces/platformStorage/PlatformDB.js — four new
methods on the interface: setAccessState(key, value, expiresAt),
getAccessState(key), deleteAccessState(key),
sweepExpiredAccessStates(now?). rqlite engine implements them;
[ACCESSSTATE] 8 conformance cases.components/messages/src/cluster_kv.js — master-held key/value
store + worker IPC primitive for the ephemeral cross-worker state
class (single-core scope only; cross-core state goes to PlatformDB).
Wire format kv:get/set/delete/clear with namespaced replies. Lazy
expire on get + 60 s sweeper. In-process fallback when process.send
isn’t available (single-process tests, CLI tools). Wired in
bin/master.js after tcpPubsub.init(). 14 unit cases under
[CLUSTERKV].components/business/src/mfa/SessionStore.js +
components/business/src/mfa/index.js — MFA session store backed on
cluster_kv instead of a per-instance Map. API turned async
(create/has/get/clear/clearAll). Same bug family as the
accessState §12: login lands on worker A, verify hits worker B → “MFA
session not found”. The header comment that said “single-core only”
reworded — under cluster.fork() “process-wide” is per-worker, not
per-core. [MT5A] cross-worker case added.components/test-helpers/src/clusterFixture.js +
components/api-server/test/clusterWorkers/accessStateWorker.js +
components/api-server/test/access-state-cluster-seq.test.js —
multi-worker test fixture (forks N children via child_process.fork,
JSON-RPC over IPC, ready handshake on boot). [XS12A/B/C] regression
cases would fail against the pre-Plan-55 in-memory Map.Two follow-ups missed when the deps bump landed; both crashed the production Docker image at boot before any config was read.
components/business/src/mfa/Profile.js +
components/business/src/mfa/SessionStore.js — swap
const { v4: uuidv4 } = require('uuid') →
const { randomUUID: uuidv4 } = require('node:crypto'). Same alias, no
call-site churn. The uuid package was dropped from package.json in the
earlier deps bump but these two MFA files still required it; first MFA-
touching require chain (api-server/methods/auth/login → mfa) crashed
MODULE_NOT_FOUND. RFC-4122 v4 byte-equivalent.components/api-server/src/server.js — move
require('backloop.dev').httpsOptionsAsync from top-level into the
if (config.get('http:ssl:backloop.dev')) block. npm install --omit=dev
skips backloop.dev because workspace promotion marks it "dev": true
in the lockfile, so the production image has no copy on disk; lazy-require
keeps the dev-loop path working while letting prod boot.A bundle of five fixes surfaced by a fresh single-core Dokku deploy with
letsEncrypt.enabled: true + embedded DNS + ACME DNS-01 wildcard.
components/business/src/acme/selfSignedPlaceholder.js — when
letsEncrypt.enabled is on and the configured http.ssl.keyFile doesn’t
exist yet, master writes a 1-day self-signed RSA-2048 cert at the
configured paths before forking workers. Workers’ https.createServer
ENOENT-races would otherwise restart-loop the cluster until ACME issued
the first cert. Real cert hot-swaps via setSecureContext when ACME
completes (existing acme:rotate IPC + reloadTls() path). CN + SAN
derived from deriveHostnames() — same hostname the eventual ACME cert
carries. Pure node-forge (already a transitive of acme-client; no new
dep).storages/engines/rqlite/src/rqliteProcess.js — -disco-mode
dns + -bootstrap-expect 1 flags now gated on a new
cluster.discoveryEnabled: true opt instead of unconditionally on
dnsDomain != null. Single-core deploys with dns.domain set (so the
embedded DNS can serve <coreId>.<domain>) no longer have rqlited block
for 30 s waiting for lsc.<domain> peers via the embedded DNS that only
starts after rqlited is ready.components/business/src/bootstrap/applyBundle.js — bootstrap
bundle now writes cluster.discoveryEnabled: true into the joiner’s
override-config.yml when the bundle ships a cluster.domain (DNS-based
multi-core). DNSless multi-core deliberately leaves the flag unset —
peers find each other via explicit core.url instead.components/dns-server/src/DnsServer.js — embedded DNS now
resolves <coreId>.<domain> from PlatformDB. Previously such queries
fell into the #answerUsername path → NXDOMAIN, leaving the hostname
advertised in hostings.*.availableCore (and used for inter-core HTTP
routing) unreachable unless the operator pre-populated
dns.staticEntries. New private branch consults
platform.getCoreInfo(prefix) between staticEntries and the username
fallback; record-emission tail extracted into #emitCoreInfoRecords and
shared with #answerUsername. Operator overrides via dns.staticEntries
still win.components/platform/src/Platform.js — coreIdToUrl() now
returns slash-terminated URLs in all branches (cache hit, derived,
dnsLess fallback). Centralized via a small withTrailingSlash() helper.
Three downstream consumers that did coreUrl + '/something' updated to
drop the leading slash and avoid double-slash:
business/src/auth/registration.js cross-core forward POST,
api-server/src/routes/reg/legacy.js ?username= redirect,
api-server/src/routes/reg/access.js pollBase (defensive — operator
may supply core:url with or without slash). Two register.js sites
that bypass coreIdToUrl() (the coreUrl || ApiEndpoint.build('', null)
fallbacks for the unknown-email and single-core/unconfigured branches)
wrapped at the call site.Dockerfile — EXPOSE now declares 80 443 3000 3001 4000
53/udp (was just 3000). Dokku’s dokku ports:add only publishes ports
the Dockerfile exposes, so native HTTPS + embedded DNS deployments needed
an explicit docker-options:add workaround. EXPOSE is informational
only — no port is actually bound until the operator publishes it.
INSTALL.md Dokku section gained a paragraph about
dokku ports:add http:80:80 https:443:443.business 354/0 (was 346, +8 new [SSPL] for the
self-signed placeholder); PG dns-server 29/0 (was 26, +3 new [DN35]
[DN36] [DN37] for the PlatformDB-resolved <coreId>.<domain> branch);
rqlite-engine [RQARGS] 18/0 (was 15, +3 covering single-core /
no-domain / discovery-without-domain); api-server PG full 967/0. Full
just test all (PG) and just test-mongo all matrix at plan close.z-schema → ajv (with z-schema-shaped error wrapper)z-schema removed from package.json dependencies. Replaced with ajv@^8 + ajv-draft-04 (our schemas use the draft-04 id: keyword) + ajv-formats.components/utils/src/jsonValidator.js — backed by ajv-draft-04, exposes the slice of the legacy z-schema API that callers depend on (validate(data, schema, callback?) either sync-returning-bool or async-via-callback, validateSchema(schema), getLastError() / getLastErrors() / lastReport). Errors are reshaped to z-schema’s wire format: { code, params: [], message, path } with z-schema-style codes (PATTERN, OBJECT_MISSING_REQUIRED_PROPERTY, INVALID_TYPE, MIN_LENGTH, MAX_LENGTH, etc.) so commonFunctions._addCustomMessage and the messages: { CODE: { code, message } } blocks in schema files keep working unchanged. required-error paths end with / (e.g. #/) to match z-schema’s exact shape so paramId fallback in _addCustomMessage resolves correctly.access.permissions(action) returns a fresh top-level object each call, with nested id: 'streamPermission'); a shared registry would error with “reference resolves to more than one schema” on the second compile.stripUnreferencedIds — drops schema-level id strings whose values aren’t $ref-targeted from anywhere in the schema. Distinguishes schema-level id: 'foo' (drop if unused) from data-property-named id: { type: 'string' } (always keep). Top-level id is preserved so self-references (e.g. systemStreamsSchema → $ref: 'systemStreamsSchema') keep resolving.components/api-server/src/schema/event.js — id-pattern regex replaced \\: (escaped colon) with plain :. ajv compiles pattern strings with the u (unicode) flag, which rejects unnecessary escapes; the colon is not a regex metacharacter and works in both modes.components/api-server/src/schema/methodError.js — subErrors.items.$ref: '#error' (id-fragment) replaced with $ref: '#' (root self-reference). ajv-draft-04 doesn’t auto-treat top-level id: 'error' as an in-document anchor; root self-ref is the portable form across both validators.require('z-schema'): components/api-server/src/schema/validation.js (the API method validator entry point), components/business/src/types.js (TypeRepository for event-type validation; both the lazy TypeValidator.validateWithSchema and the eager TypeRepository._validator), components/api-server/test/helpers/validation.js (test-side response-shape assertions). Callers consume the wrapper via const { jsonValidator } = require('utils'); const v = jsonValidator().just test all → 1742 / 0; Mongo just test-mongo all → 1735 / 0 (PG-pool exhaustion flake during cross-engine matrix runs cleared after pg_ctl restart — environmental, not from this slice).mongodb driver 4.17 → 7.2 bumpmongodb bumped from ^4.11.0 to ^7.2.0. Three majors of driver: v5 removed callback-style APIs entirely (Promise-only), v6 dropped legacy findOneAndUpdate { value: doc } wrapper (returns the doc directly), v7 ships BSON v7 + new connection-string parser + @mongodb-js/saslprep.storages/engines/mongodb/src/Database.js — every collection method (findOne, find().toArray(), insertOne/Many, updateOne/Many, findOneAndUpdate, deleteOne/Many, countDocuments, drop, listIndexes, dropDatabase) wrapped via two new local helpers p2c(promise, callback) / p2cWithDup(promise, callback). The Database class still exposes its callback-shaped public API to consumers (storages/business/api-server) — only the driver-facing internals changed. findOneAndUpdate now returns the doc directly: dropped the r && r.value indirection. The connection bootstrap no longer issues db('admin').command({ setFeatureCompatibilityVersion: '6.0' }) — server FCV is an operator concern, not application init (and v7’s confirm: true requirement breaks against older servers).connectTimeoutMS, socketTimeoutMS, writeConcern: { j, w }, appname) all forward-compatible.mongodb-core was a stale devDependency with zero consumers — left in the file for now (separate cleanup if anyone touches it).just test all → 1742 / 0; Mongo just test-mongo all → 1735 / 0 (one PG-pool exhaustion flake during the cross-engine run sequence, not a regression — cleared after a pg_ctl restart -m fast).bluebird from production runtimebluebird from root package.json dependencies. 26 production files migrated. Pass 1 (8 sites) replaced bluebird.try / bluebird.all / bluebird.map / bluebird.mapSeries with native equivalents (Promise.all, Promise.all(arr.map(fn)), for-of + await). Pass 2 (74 sites) replaced bluebird.fromCallback((cb) => fn(args, cb)) with a tiny in-tree helper fromCallback exposed from components/utils/.components/utils/src/fromCallback.js — 9-line wrapper that turns a (cb) => ... thunk into a Promise resolving with the callback’s value (or rejecting with the callback’s err). Identical semantics to bluebird’s fromCallback. Exported as utils.fromCallback.bluebird from dependencies to devDependencies — three test files (components/storage/test/hook.js, components/hfs-server/test/support/child_process.js, storages/engines/mongodb/test/hook.js) still import it for legacy promise wiring; migrating them is a test-infra refactor for later (or folds into TS+ESM). npm ls bluebird --omit=dev shows no direct production dep; bluebird@3.7.2 survives only via email-templates → consolidate (out of our control).components/test-helpers/src/helpers-base.js — dropped the no-op bluebird: require('bluebird') re-export. Zero consumers grep-confirmed.just test all → 1742 / 0; Mongo just test-mongo all → 1735 / 0 (one transient [WHBK] [BLNP]/[1VIT] webhook-retry-state flake on a single matrix run, cleared on re-run — same family as the documented [WH01] flakes; not caused by this slice).async (callback control-flow lib) from production runtimeasync.series / forEachSeries / forEachOfSeries / until to native async/await + for-of/while loops. Affected: components/api-server/src/API.js (forEachSeries → manual runNextMethod chain to preserve tracing+error semantics), components/api-server/src/Result.js (forEachOfSeries → nextElement chain), components/api-server/src/methods/accesses.js (forEachSeries + nested series → IIFE async/await), components/api-server/src/methods/events.js (series → linear async/await), components/api-server/src/methods/profile.js (series → callback chain), components/hfs-server/src/metadata_cache.js (series of mixed sync+promise → linear async/await; deleted unused toCallback helper), components/test-helpers/src/{InstanceManager,DynamicInstanceManager}.js (until → while loop), components/test-helpers/src/data.js (5 series sites → IIFE async/await; introduced tiny runSeries helper for dumpCurrent/restoreFromDump which mix callback-style step lists).async from dependencies to devDependencies. Several *-seq.test.js files still use async.series / async.eachSeries; migrating those is a test-infra refactor we can do alongside the TS+ESM conversion.npm ls async --omit=dev shows no direct dep; async@3.2.6 remains as a transitive of nconf (boiler) and winston — out of scope here.just test all → 1742 / 0; Mongo just test-mongo all → 1735 / 0.cuid → @paralleldrive/cuid2 for production ID minting@paralleldrive/cuid2@^3.3.0 to dependencies. Moved cuid@^2.1.8 from dependencies to devDependencies — test-helpers still uses cuid.slug() (cuid2 has no .slug() equivalent) so cuid is kept as a dev-only dep.require('cuid') to const { createId: cuid } = require('@paralleldrive/cuid2') (or createId: generateId where the local alias was generateId). All call sites already use cuid() (default 24-char form) — cuid2’s createId() is a clean drop-in.components/api-server/src/schema/event.js — id-format pattern broadened to accept three alternatives: system-stream id (:scope:name), legacy cuid v1/v2 (^c[a-z0-9-]{24}$), and cuid2 (^[a-z][a-z0-9]{23}$). The legacy pattern stays because existing IDs in databases are still cuid v1/v2 strings; the new pattern is required because cuid2 IDs don’t share the c… prefix.c prefix (cuid2 format), versus the prior c[a-z0-9-]{24} (25 chars total) cuid v1/v2 format. Existing IDs in production databases remain valid (string columns; no migration). Clients that regex-validate IDs against the legacy ^c[…] pattern will need updating; the relaxed schema regex above accepts both.just test all → 1742 / 0; Mongo just test-mongo all → 1735 / 0.lru-cache 7.14 → 11.0; cron 2.4 → 4.4lru-cache bumped from ^7.14.1 to ^11.0.0. The default export is now { LRUCache } (renamed in v8). Six call sites updated with the alias trick const { LRUCache: LRU } = require('lru-cache') so the existing new LRU({ … }) constructions stay verbatim. Affected: components/cache/src/index.js, components/hfs-server/src/metadata_cache.js, components/hfs-server/src/web/op/store_series_batch.js, storages/engines/postgresql/src/AuditStoragePG.js, storages/engines/sqlite/src/userAccountStorage.js, storages/engines/sqlite/src/userSQLite/Storage.js. Constructor options (max, ttl, dispose(value, key)) are forward-compatible.cron bumped from ^2.4.4 to ^4.4.0. v4 changed the constructor: new CronJob({ cronTime, onTick }) no longer works (the constructor is positional in v4); use CronJob.from({ cronTime, onTick }) instead. Single call site updated in components/previews-server/src/routes/event-previews.js. Cron pattern format unchanged (6-field with seconds slot still supported).just test all → 1742 / 0; Mongo just test-mongo all → 1735 / 0.components/tracing/src/Tracing.js — collapsed to a single DummyTracing no-op class. The exported Tracing and DummyTracing symbols both now point at the same no-op. The architectural slot is preserved so a future tracer (e.g. an OpenTelemetry adapter) can plug in here without touching consumers.components/tracing/src/index.js — dropped the isTracingEnabled / launchTags config branches; initRootSpan always returns a DummyTracing instance. tracingMiddleware simplified to (req, res, next) => { req.tracing ??= new DummyTracing(); next(); }.components/tracing/src/databaseTracer.js — replaced the Jaeger-driven monkey-patcher with module.exports = function patch () {};. Callers in components/storage/src/index.js and storages/index.js need no edits.components/tracing/src/HookedTracer.js — replaced with a no-op HookedTracer class.components/hfs-server/src/tracing/cls.js — replaced with a no-op Cls class. setRootSpan/getRootSpan/startExpressContext all return null or pass through.components/hfs-server/src/tracing/middleware/trace.js — passthrough that calls next().components/hfs-server/src/application.js — dropped opentracing and jaeger-client imports; produceTracer removed; replaced with an inline NoopTracer / NoopSpan minimal stub used by Context#childSpan.components/hfs-server/src/server.js — removed the if (traceEnabled) block that registered the trace+cls middleware. The traceEnabled config flag and the clsWrapFactory / tracingMiddlewareFactory imports are gone.components/hfs-server/src/web/controller.js — storeErrorInTrace no longer reads opentracing.Tags.ERROR; it tags the root span (now always null) with the literal string 'error'. With cls returning null, the function early-returns; behaviour is unchanged from the prior trace.enable: false default.package.json dependencies:
jaeger-clientcls-hookedopentracingshimmernpm ls jaeger-client opentracing cls-hooked shimmer --omit=dev is empty.components/tracing/src/Tracing.js.components/tracing/. Operators using New Relic see no change. Operators relying on trace.enable: true (none we’re aware of) will find the flag is now ignored — Jaeger is gone.just test all → 1742 / 0; Mongo just test-mongo all → 1735 / 0.hjson from package.json. Zero call sites in the entire repo (production or test).url from package.json. The single require('url') site in components/test-helpers/src/spawner.js resolves to the Node 24 built-in url module (same name) — the npm package was a no-op shadow.mkdirp from package.json. Replaced 8 call sites across 6 production files + 1 test-helper file with fs.mkdir(path, { recursive: true }) / fs.mkdirSync(path, { recursive: true }) (Node ≥ 10). Affected: components/business/src/integrity/MulterIntegrityDiskStorage.js, components/previews-server/src/attachmentManagement.js, components/storage/src/userLocalDirectory.js, storages/engines/filesystem/src/EventLocalFiles.js, storages/engines/sqlite/src/usersLocalIndex.js, storages/engines/rqlite/src/rqliteProcess.js, components/test-helpers/src/data.js.body-parser from package.json. Replaced 3 production sites + 3 test sites with the express-built-in equivalents (express.json() / express.urlencoded()) — Express 4.16+ ships them. Affected: components/api-server/src/expressApp.js, components/hfs-server/src/server.js, components/previews-server/src/expressApp.js, plus three local-HttpServer test mocks.awaiting, fs-extra, backloop.dev, msgpack5 from dependencies to devDependencies:
awaiting is required by 3 acceptance test files and zero production files.fs-extra is required by 1 storage test file and zero production files.backloop.dev is loaded only behind the http:ssl:backloop.dev config flag in components/api-server/src/server.js, a local-dev convenience; production runs use ACME (Plan 35) or operator certs.msgpack5 is required by components/test-helpers/src/{child_process,spawner}.js only.components/tracing/ remains a real production dependency (8 hot-path call sites) even when Jaeger is disabled, and that trace.enable: false only short-circuits the Tracing body — wiring is hot-path. Future deletion of components/tracing/ requires touching all 8 callers in the same patch (filed as XXX-Backlog/PLAN52-PHASE4-TRACING-RIPOUT.md).just test all → 1742 / 0; Mongo just test-mongo all → matches Plan 52 Phase 3.S.2 baseline.async callback-control-flow lib (XXX-Backlog/PLAN52-PHASE4-ASYNC-CALLBACK-DROP.md), drop bluebird (recommended to fold into TS+ESM migration), replace unix-timestamp’s duration DSL, major bumps for lru-cache / cron / slug / email-templates / nodemailer (_plans/XX-deps-major-bumps-later/PLAN.md), mongodb 4→7 (own plan TBD), z-schema → ajv (own plan TBD), cuid → cuid2 (own plan TBD).superagent → native fetch complete; superagent moved to devDependenciescomponents/api-server/src/methods/helpers/mailing.js — _sendmail() uses native fetch. Callback contract preserved (cb(err, res)); parseError() now also matches ENOTFOUND/ECONNREFUSED in the unreachable-endpoint branch since native fetch’s reject messages differ from superagent’s.components/business/src/mfa/Service.js — _makeRequest() uses native fetch, JSON-encoding non-string POST bodies and explicitly throwing on !res.ok so the existing try/catch → invalidOperation('mfa-sms-provider-error') translation still fires. Consumers (SingleService, ChallengeVerifyService) await without reading the response body, so the swap is transparent at call sites.nock bumped from ^13.2.9 to ^14.0.13 (latest stable). v14’s headline feature is native fetch interception via @mswjs/interceptors, which is what unblocked the two swaps above. Engine constraint >=18.20.0 <20 || >=20.12.1 is satisfied by Node 24. No test API surface change required — nock(host).post(...).reply(...) chain works identically.components/api-server/test/mfa-seq.test.js — nock.enableNetConnect('127.0.0.1') widened to enableNetConnect(/127\.0\.0\.1|localhost/). nock v14 intercepts native fetch too, and the rqlite client (DBrqlite.query/execute) connects to localhost:4001 — '127.0.0.1' and 'localhost' are not aliased by the allowlist.superagent moved from dependencies to devDependencies (still needed by components/test-helpers/src/{request,parallelTestHelper}.js). Production runtime no longer pulls superagent — and therefore no longer pulls its transitive formidable@2.1.5. npm ls formidable --omit=dev is now empty; formidable survives only via the test surface.just test all → 1742 / 0; Mongo just test-mongo all → 1734 / 0 (one pre-existing flake [AUTH] [AU01] [FMJH] on concurrent login race, not caused by this slice — re-runs cleanly).superagent call sites are now on native fetch). Phase 3.F (formidable cleanup) auto-closed: production dep graph is formidable-free.superagent → native fetch for business/types.js and business/webhooks/Webhook.jscomponents/business/src/types.js — TypeRepository.tryUpdate() now fetches the remote event-types definition via Node’s native fetch instead of superagent. Throws an explicit Error("Event types fetch failed: HTTP <status> <statusText>") on non-2xx so the existing try/catch → unavailableError(err) path still triggers. No behavior change at the call sites.components/business/src/webhooks/Webhook.js — makeCall() uses native fetch. To preserve the prior superagent semantics consumed by runOnce() and the webhooks.test API method, makeCall() now explicitly throws on !res.ok with err.response = { status } attached; native fetch does not throw on 4xx/5xx by default. Removed the unused request = require('superagent') import.components/api-server/src/methods/helpers/mailing.js and components/business/src/mfa/Service.js still use superagent. Both call sites are exercised by tests that intercept HTTP via nock@^13.5.6, which does not intercept Node 24’s native fetch (Undici dispatcher). Migrating these two requires either upgrading to nock@^14 (native fetch interceptor) or switching the affected tests to a real local HTTP server. Tracked in the next Phase 3.S.2 slice; out of scope here.superagent therefore stays in runtime dependencies for now. The two completed swaps still reduce the production runtime’s reliance on it.just test all → 1742 / 0; Mongo just test-mongo all → 1735 / 0 (both match Phase 3.L baseline).@pryv/boiler vendored as an in-tree workspace packagecomponents/boiler/ workspace package — exact copy of the @pryv/boiler@1.2.4 source tree (8 files, 4 src/ files + lib/nconf-yaml + README + LICENSE + package.json). Resolves under the existing npm-workspace symlink at node_modules/@pryv/boiler so every require('@pryv/boiler') call site continues to work unchanged.package.json — @pryv/boiler removed from runtime dependencies; the workspace package now satisfies the import. No longer pulls boiler from the upstream pryv/pryv-boiler.git#semver:^1.2.4 git URL at install time.package-lock.json — boiler’s transitive deps (debug, js-yaml, nconf, superagent, winston, winston-daily-rotate-file) now resolve against the in-tree workspace; root-level entries unchanged in production behaviour.just test all (PG default) → 1742 / 0 (matches pre-vendoring baseline).superagent path, the unused notifyAirbrake/airbrake stubs, and the pluginAsync ordering surface in follow-up commits without coupling those changes to a package.json dep change. Each simplification step is a standalone commit with its own test pass.storages/engines/rqlite/scripts/setup — replaced $0 with ${BASH_SOURCE[0]} for SCRIPT_FOLDER resolution. The script is sourced (not exec’d) from scripts/setup-dev-env, which made $0 resolve to the parent script’s directory. As a result REPO_ROOT=$SCRIPT_FOLDER/../../../.. landed one parent above the actual repo root, and bin-ext/rqlited was installed outside the repo. The start script (which uses its own correct path resolution) then could not find the binary, rqlited never came up, and every test that touches PlatformDB failed with TypeError: fetch failed → ECONNREFUSED against localhost:4001. Masked since 2026-04-14 by continue-on-error: true on the test jobs.storages/engines/mongodb/scripts/setup — same one-line fix for consistency. The latent bug did not manifest for mongo because mongo’s setup uses $VAR_PRYV_FOLDER (exported correctly by the parent) for path computation rather than SCRIPT_FOLDER.storages/engines/postgresql/src/userAccountStorage.js — getPasswordHash() now returns undefined (not null) when no password row exists, matching the conformance contract and the MongoDB engine. getCurrentPasswordTime() now throws Error("No password found in database for user id ...") when no row exists, matching the MongoDB engine’s behaviour. Closes the three pre-existing PG-side [UAST] conformance failures ([V54S] + the clearHistory() and _clearAll() round-trip checks)..github/workflows/ci.yml — test-mongo job removed. PostgreSQL is the default baseStorage engine since 2026-04-24; MongoDB is opt-in (just test-mongo all) and validated locally rather than in CI. continue-on-error: true stopgap removed from test-postgres; the job is fully blocking again. docker job depends only on test-postgres + lint.AGENTS.md at repo root — fast-orientation guide for LLM coding agents (Claude Code, Cursor, Copilot, etc.) bootstrapping against open-pryv.io v2. Covers the “single-binary codebase” framing, annotated repo map, local-run + test commands, five architectural truths (master.js lifecycle, native TLS, wildcard certs via deriveHostnames, pluggable storage engines, cluster CA lifecycle), common pitfalls, config precedence, and a curated block of in-repo + pryv.github.io links.README.md — “For LLM coding agents” paragraph at the bottom points at AGENTS.md.just dev / just test-postgres recipes, wrong engine-config YAML keys, stale meta-repo framing, outdated README-DBs.md warning). All such issues fixed; file length 218 lines, under the 250-line soft cap.components/mail/ workspace package — ports Sender / Template / errors from the standalone service-mail repo; adds TemplateRepository against an injected templateExists (so the backing store can be tmp-dir, disk or PlatformDB) and a tmp-dir-materialize emailTemplatesDelivery adapter around the email-templates npm module. Façade init() / isActive() / send() / refresh() / close() with silent no-op before init so callers don’t need to guard.components/mail/src/TemplateSeeder.js — idempotent seedIfEmpty({platformDB, templatesRootDir}). Walks <root>/<type>/<lang>/*.pug and populates PlatformDB only when zero mail-template/* rows already exist.storages.init() when services.email.method === 'in-process'. Try/catch guard: a malformed templatesRootDir never blocks master startup.setMailTemplate / getMailTemplate / getAllMailTemplates / deleteMailTemplate(type, lang, part?). Keyspace mail-template/<type>/<lang>/<part> on the existing rqlite keyValue table. deleteMailTemplate(type, lang) with no part wipes both html + subject scoped to that <type>/<lang>/ prefix only.components/api-server/src/methods/helpers/mailing.js — new 'in-process' case in the method switch. First call in a worker lazy-inits the mail façade with storages.platformDB.getAllMailTemplates + the per-core SMTP config. Callback contract preserved — existing callers (registration.js::sendWelcomeMail, account.js reset-password flow) don’t need any edit.bin/mail.js CLI + /system/admin/mail/* routes (see CHANGELOG-v2.md). Write routes emit process.send({type:'mail:template-invalidate'}) so master broadcasts the nudge to every sibling worker (including the originating one is skipped); each worker’s components/mail/src/index.js subscribes via process.on('message', …) in init() and calls refresh() on receipt.cluster.on('message', …) case for mail:template-invalidate; broadcasts to all workers except the originator.email-templates@^10.0.1, nodemailer@^6.9.16, pug@^3.0.4 added as production deps on the root package.json. No transitive conflicts with the existing stack.[MAILTMPL] 7 cases on components/platform/test/conformance/PlatformDB.test.js — round-trip, null-absent, overwrite, bulk decode, single-part delete, lang-wide delete scoped, namespace isolation from dns-record/* / user-core/* / observability/*.[MAILSEND] / [MAILTMPL] / [MAILREPO] / [MAILADAPT] / [MAILFCD] / [MAILSEED] — 21 unit tests under components/mail/test/.[MLIP] 2 cases on components/api-server/test/methods/helpers/mailing.test.js — end-to-end Pug render + nodemailer jsonTransport dispatch via the helper.[MAILCLI] 9 subprocess cases + [MAILADM] 9 HTTP cases on components/api-server/test/./app/bin-ext/Dockerfile — rqlited binary relocated from /app/var-pryv/rqlite-bin/rqlited → /app/bin-ext/rqlited. Operators who bind-mount /app/var-pryv (intending to persist rqlite data) no longer shadow the baked-in binary. The only persistent path docker operators need is /app/var-pryv/rqlite-data, now declared as VOLUME.var-pryv/rqlite-bin/rqlited → bin-ext/rqlited in the setup script, start script, rqlite manifest default, bin/master.js fallback, and two test files that hard-coded the path. .gitignore covers the new location.storages.engines.rqlite.binPath in override-config.yml are unaffected either way.INSTALL.md — new “Docker / Dokku deployment” section with a “What to persist” checklist, Dokku-specific storage mount commands, and an explicit note about dokku ps:restart requiring dokku proxy:build-config <app> afterward (nginx upstream caching bug that doesn’t refresh on restart). Also documents the DATABASE_URL-not-auto-consumed caveat and the UDP/53 docker-options workaround for DNS-active multi-core on Dokku.config/default-config.yml — storages.base.engine is now postgresql (was mongodb). Mongo remains fully supported; set storages.base.engine: mongodb in override-config.yml (or export STORAGE_ENGINE=mongodb for tests) to pick it explicitly. Deployments that pin the engine in override-config.yml are unaffected.justfile — just test + just test-parallel + all other test-* recipes run PG by default. New just test-mongo / just test-mongo-parallel recipes for the Mongo path. Removed: test-pg, test-pg-parallel (now redundant)..github/workflows/ci.yml — test-postgres job runs just test all, test-mongo job runs just test-mongo all.components/business/src/observability/ — provider-agnostic façade. isActive() / setTransactionName / recordError / recordCustomEvent / startBackgroundTransaction. Every provider call wrapped in try/catch so observability can never break a request.components/business/src/observability/logForwarder.js — wraps a boiler logger to mirror its level methods into observability.recordError / recordCustomEvent. Errors always go to the provider’s Error inbox regardless of log level; warn/info/debug become PryvLog custom events queryable via NRQL.components/business/src/observability/providers/newrelic/{boot,adapter,newrelic.config.template}.js — thin wrapper over the newrelic npm package. Agent config is driven entirely by env vars the master process populates, so no on-disk config edits are required per deployment.bin/_observability-boot.js — must be require()d first in every entrypoint. Bypasses in NODE_ENV=test or when PRYV_OBSERVABILITY_PROVIDER is unset; otherwise dispatches to the provider’s boot module so the underlying agent loads before http / express / pg / etc.setObservabilityValue / getObservabilityValue / getAllObservabilityValues / deleteObservabilityValue. Keyspace observability/<key> in the existing rqlite keyValue table — no schema change.getObservabilityConfig() merges local YAML override + PlatformDB rows + derived fields (hostname from new URL(core.url).hostname, appName fallback). Local observability.enabled: false always wins; otherwise PlatformDB is authoritative. Secret rows decrypted on demand via AtRestEncryption with HKDF-derived keys (source: auth.adminAccessKey, per-key purpose label).platform.getObservabilityConfig() before forking workers, builds a shared observabilityEnv object, and spreads it into every cluster.fork({...}) call (api / hfs / previews). Environment variables include PRYV_OBSERVABILITY_PROVIDER, NEW_RELIC_LICENSE_KEY, NEW_RELIC_APP_NAME, NEW_RELIC_PROCESS_HOST_DISPLAY_NAME, NEW_RELIC_LOG_LEVEL, NEW_RELIC_HIGH_SECURITY=true, NEW_RELIC_HOME.bin/observability.js — storages barrel directly, no HTTP. Parses --help before boiler init (same pattern as bin/dns-records.js).storages/engines/rqlite/test/platformdb-conformance.test.js — 6 new [RQPF] cases under the shared components/platform/test/conformance/PlatformDB.test.js: round-trip, overwrite/rotation, bulk read, delete, namespace isolation vs dns-record/* and user-core/*.components/api-server/test/observability-seq.test.js — [OBS] suite (9 cases): Platform round-trip with encryption, local enabled:false override wins, appName fallback, hostname derivation, façade no-op when no provider, shim NODE_ENV=test bypass, shim unset-env no-op, logForwarder errors-only default, logForwarder warn level forwards errors + warns.Surfaced during pryv.me v2 rollout. The items below make cross-core registration atomic, expose the SDK-expected shape of /service/info + /reg/access, and fix several subtle multi-core plumbing bugs that appeared once a real two-core deployment hit a freshly-delegated domain.
Previously, a POST /users landing on a core whose core.id didn’t match the user’s chosen hosting would call Platform.validateRegistration, which reserved unique fields + wrote user-core/<username> in PlatformDB, then returned {core: {url: targetCoreUrl}} for the client to re-POST. Non-compliant SDKs silently swallowed the redirect, stranding orphaned user-core rows and empty PG on the target core.
components/business/src/auth/registration.js — new forwardIfCrossCore step inserted into the auth.register chain between prepareUserData and validateOnPlatform. Calls platform.selectCoreForRegistration(hosting); if target ≠ self, HTTPS-POSTs the original body to {targetUrl}/users (the target’s own forwardIfCrossCore is idempotent when target == self). Target’s response (minus its own meta block) is merged into result.forwarded. Atomic on the target: unique-field reservation, user-core assignment, user insert, welcome mail all on one core.validateOnPlatform, createUser, buildResponse, sendWelcomeMail all short-circuit on result.forwarded — no duplicate work, no duplicate mail.components/platform/src/Platform.js — validateRegistration(username, invitationToken, uniqueFields, hosting) now takes + honours the caller-provided hosting. Previously it always called selectCoreForRegistration() without the hosting filter, so with a least-users tiebreak a new aws-us-east-1 registration could leak to aws-eu-central-1 just because the latter had fewer users.components/api-server/src/methods/auth/register.js — wires forwardIfCrossCore into the auth.register method chain./service/info multi-core shapecomponents/api-server/src/schema/service-info.js — added optional version field.components/api-server/src/methods/service.js — populates version from getAPIVersion(). lib-js + app-web-auth3 gate on version >= 1.6.0 to pick the direct-core /users registration endpoint. Before this, our /service/info had no version → SDKs fell back to the legacy /reg/user via reg.{domain} round-robin, which (before the forward fix) compounded the orphaned-user-core bug.config/plugins/public-url.js — in multi-core (dnsLess.isActive: false) mode, register: https://reg.{domain}/ and access: https://access.{domain}/access/ instead of the old register: https://core-{id}.{domain}/reg/. The reserved-subdomain URLs are core-symmetric and match the v1 Pryv.io URL shape; regSubdomainPathMap middleware (below) handles the /reg prefix inside the core.config/plugins/config-validation.js — new REQUIRED_SERVICE_FIELDS = ['name', 'serial', 'home', 'support', 'terms', 'eventTypes'] check. Master fails fast with a clear error instead of starting into an api-server crash-loop when operators forget the service: block.bin/master.js — added config-validation plugin to master’s boiler init (previously only in api-server’s application.js), so the service-required-fields check triggers on master bring-up too.components/dns-server/src/DnsServer.js — new RESERVED_SERVICE_NAMES = ['reg', 'access', 'mfa']. The embedded DNS auto-resolves these three subdomains to every available core’s IP (via getAllCoreInfos()), no dns.staticEntries required. Operators still own sw, mail, etc. via staticEntries; documented in config/default-config.yml.components/api-server/src/expressApp.js — two multi-core-only middleware additions:
subdomainToPath’s ignoredSubdomains list now includes reg, access, mfa, and every key from dns.staticEntries. Without this, access.pryv.me (6 chars, matches the username regex) was rewritten to /access/… and fell into the username router.regSubdomainPathMap middleware: when Host: reg.{domain} (or access. / mfa.), prepend /reg to req.url before route matching. Lets clients use rootless v1-style URLs (reg.pryv.me/perki/server, reg.pryv.me/service/info) while the internal routing stays under /reg/*. Idempotent — skips when the path already starts with /reg/.components/api-server/src/routes/register.js — when dnsLess.isActive: false, also expose GET /service/info at the root (alias for /reg/service/info). Lets SDKs bootstrap from https://reg.{domain}/service/info directly.components/api-server/src/routes/reg/legacy.js — GET + POST /reg/:uid/server now look up the core via platform.getUserCore() (PlatformDB, replicated) instead of usersRepository.usernameExists() (per-core SQLite index). Without this, round-robin DNS on reserved subdomains caused 50 % 404s because only the user’s home core had them in its local index. getCoreUrlForUser returns null when no mapping exists so the handler 404s cleanly./reg/access (auth popup) shapecomponents/api-server/src/routes/reg/access.js — POST /reg/access response now includes:
authUrl (primary) — built from access.defaultAuthUrl + query params (lang, key, requestingAppId, poll, poll_rate_ms, serviceInfo). SDKs open this URL in the sign-in popup.url (deprecated) — same value, kept for v1 SDK compatibility.poll — core-affine URL built from core.url, not the cluster-wide service.register. The poll state is in-memory per core, so a poll GET must pin to the same core that served the POST; using service.register round-robined across cores and caused unknown-access-key on half the polls.lang, returnURL + returnUrl (camelCase lib-js expects), serviceInfo (v1-compatible — SDKs re-hydrate from the body)./reg/access/:key NEED_SIGNIN response now mirrors the same fields (poll, authUrl, url, lang, returnUrl/returnURL, serviceInfo) so app-web-auth3’s context.init() → setServiceInfo(accessState.serviceInfo) doesn’t crash with “Cannot read properties of undefined (reading ‘name’)” and clients re-hydrating state from the poll body see the poll URL.state.pollUrl + state.authUrl are stashed on the in-memory access state at POST time so the subsequent GETs echo them verbatim.bin/master.js — cluster.setupPrimary({ args: process.argv.slice(2) }) before forking workers. cluster.fork() by default runs the worker with only [node, master.js] — argv after the script name is silently dropped. Deployments that layered --config host-config.yml had their workers fall back to NODE_ENV-based config and silently use the wrong storage engine / ports.components/middleware/src/project_version.js — process.mainModule || require.main || module fallback. process.mainModule was deprecated and can be undefined in Node 22 when the entrypoint is loaded via a wrapper or cluster fork; the old code threw TypeError: Cannot read properties of undefined (reading 'paths') which was swallowed by boiler’s file logger and surfaced as silent api-server worker crash loops.components/api-server/bin/server — catch-block mirrors fatal errors to process.stderr. Master’s api worker died (code=1, signal=null) now always has an actionable cause attached instead of being silent.components/business/src/acme/CertRenewer.js + AcmeOrchestrator.js + bin/master.js — PlatformDBDnsWriter accepts an optional dnsServer and calls dnsServer.refreshFromPlatform() immediately after writing _acme-challenge.<zone> TXT to PlatformDB. Previously relied on the DnsServer’s 30 s periodic refresh, so LE’s DNS-01 validator often failed with “No TXT records found”. AcmeOrchestrator.build() threads dnsServer through; bin/master.js passes it. Real LE wildcard issuance on a fresh cluster now succeeds on the first attempt instead of 15–30 min after rqlite caught up.components/business/src/auth/registration.js::sendWelcomeMail — guards against missing services.email in config (fresh bundle-bootstrapped cores have no default) and against forwarded registrations (target core already sent the mail). Before, a missing services.email threw Cannot read properties of undefined (reading 'enabled') AFTER createUser had already persisted the user, leaking a 500 response to the client even though the registration had technically succeeded.Latent bug since the v2 snapshot — only visible on a cluster that runs under NODE_ENV=production with a production-config.yml that does not re-declare custom:systemStreams:account. On the pryv.me cluster this surfaced as welcome-mail failing with recipient.email = undefined despite POST /users carrying email in the body and returning 201.
Root cause: @pryv/boiler loads default-config.yml AFTER running synchronous plugin extras, but BEFORE awaiting pluginAsync extras (via config.initASync()). The systemStreams plugin reads config.get('custom:systemStreams:account') and builds accountMap + accountFields. When registered as plugin (sync), it ran against a config that still had no custom.* block, so accountMap was missing :system:email, User.loadAccountData never copied params.email → user.email, and registration.js::sendWelcomeMail saw undefined. In dev/test this was hidden because {development,test}-config.yml declare custom.systemStreams.account in the base scope (loaded before sync plugins).
Fix: 16 occurrences of { plugin: require('.../config/plugins/systemStreams') } changed to { pluginAsync: require(...) }. pluginAsync.load(config) is awaited in initASync() (boiler config.js:220), after default-config.yml loads at line 156. All downstream code that reads config.get('systemStreams') (notably accountStreams.init() via await getConfig() in components/business/src/system-streams/index.js) already awaits configInitialized, so no race.
Files touched: bin/{master,bootstrap,migrate,backup,dns-records,integrity-check}.js, components/api-server/src/application.js, components/webhooks/src/application.js, components/hfs-server/src/application.js, components/previews-server/src/{server,runCacheCleanup}.js, components/api-server/test/helpers/core-process.js, components/test-helpers/src/api-server-tests-config.js, components/test-helpers/scripts/dump-test-data.js, components/webhooks/test/test-helpers.js, components/hfs-server/test/acceptance/test-helpers.js.
Test matrix re-verified after the switch — PG 1654/0, Mongo 1676/0. No test asserts a specific accountFields order that would have flipped with the new merge behaviour.
${VAR} placeholdersproduction-config.yml uses shell-style ${PRYV_LOGSDIR} / ${PRYV_DATADIR} placeholders in path values, but nothing in the boiler/nconf stack actually expands them. When the env var was unset at NODE_ENV=production (e.g. a stray bin/server run during live debugging), Winston’s file transport treated the literal string as a path and created a directory named ${PRYV_LOGSDIR} on disk.
Fix: config/plugins/config-validation.js::checkIncompleteFields now matches \$\{([A-Z_][A-Z0-9_]*)\} in every string value alongside the existing REPLACE sentinel scan. Unresolved placeholders fail startup with a clear error naming the missing env var. Same active: false / enabled: false block-skip rules apply. .gitignore also picks up the literal ${PRYV_LOGSDIR} / ${PRYV_DATADIR} names so an accidental stray dir doesn’t pollute git status.
user-core/* rows from register/servers.jsonl.gzstorages/interfaces/backup/FilesystemBackupReader.js — new readServerMappings() method that streams {username, server} rows from register/servers.jsonl[.gz]. No-op when the register/ subdir is absent (open-pryv.io v1.9 or v2→v2 backups).storages/interfaces/backup/BackupReader.js — default readServerMappings() on the base interface yields an empty async iterator, so sources without register data (any reader that doesn’t override it) inherit a safe default.components/business/src/backup/RestoreOrchestrator.js — _restorePlatform now also iterates readServerMappings(); for each mapping, writes a user-core/<username> row to PlatformDB. Maps the v1 server hostname (e.g. “co1.pryv.me”) to whichever core is the SOLE available core on the destination — the common case for single-core restore. Multi-core destinations with more than one available core are left as a no-op for now; a future pass can accept a --core-map option. Previously v1→v2 restores left every user’s DNS resolution broken until the operator manually INSERTed user-core/* rows.components/api-server/test/reg-multicore-dnsless-false-seq.test.js (new) — regression suite covering the cross-core forward, /reg/access POST+GET shape, /service/info required fields + version + reserved subdomains, and the v1→v2 register-mappings restore path. Uses a targeted global.fetch interceptor (passes through to real fetch except for the inter-core forward URL) so the rqlite HTTP client keeps working during the test.components/api-server/test/reg-multicore-seq.test.js [MC01A/B] — rewritten from “must return redirect” to “HTTPS-forwards POST to target + atomic on failure” to match the new behaviour; same targeted-fetch interceptor.components/api-server/test/service-info.test.js [FR4K] — tolerates the new version field and the response-envelope meta block.components/dns-server/test/dns-server.test.js [DN11] — asserts reserved subdomain reg.{domain} resolves to A records (all core IPs), not CNAME.components/cache/test/acceptance/cache.test.js [FELT] — this.retries(3) on the 15%-cache-gain timing assertion. The thresholded comparison was flaky under scheduler noise on shared dev VMs (5–15 % gain range); retries turn transient noise into eventual success without weakening the signal.Surfaced when running the full matrix against the distribution changes above. Changes are small, isolated, and carry no API behaviour impact.
config/plugins/config-validation.js — checkIncompleteFields now skips the REPLACE sentinel scan on any block where active === false or enabled === false. Fixes dead-code if (obj.active && !obj.active) return (always false). Unblocks default-config placeholders like letsEncrypt.email: 'REPLACE ME' / letsEncrypt.atRestKey: 'REPLACE ME' (operators only set these when letsEncrypt.enabled: true) from tripping startup in vanilla config.components/api-server/src/application.js — config-validation is now registered as pluginAsync (previously plugin). Required because serviceInfo (scope loaded from serviceInfoUrl) is itself async; the validator’s required-service-fields check would otherwise fire before service.* was populated and always fail-fast with “service fields missing”.components/api-server/src/methods/service.js — removed the first-call this.serviceInfo cache. Service info is now read live from config every request. The cache leaked state between tests sharing a single api-server and would also prevent future runtime service: updates (e.g. admin-API edits) from being visible without a restart.components/api-server/src/routes/reg/legacy.js — getCoreUrlForUser in single-core mode now verifies the user exists (usersRepository.usernameExists()) before returning the core URL. Previously any arbitrary username would resolve to the local URL, shadowing the 404 the /reg/:uid/server routes are supposed to emit for unknown users.storages/interfaces/migrations/ — contracts + conventions for forward-only, timestamp-ordered schema migrations. migration.d.ts defines the { up, down? } shape; MigrationRunner.d.ts defines the runner + MigrationCapableEngine contract; README.md captures the model (integer version +1 per migration, YYYYMMDD_HHMMSS_<slug>.js filenames, idempotency requirement, per-engine schema_migrations storage).storages/interfaces/migrations/MigrationRunner.js — runtime. discoverMigrations() walks an engine’s migrations/ dir and lex-sorts; status() reports per-engine { currentVersion, pending }; runAll({ targetVersion, dryRun }) applies up() in order and bumps version via the engine’s setVersion(). createMigrationRunner() auto-wires from the active storages barrel, iterating engines that export getMigrationsCapability().storages/engines/postgresql/src/SchemaMigrations.js — lazy CREATE TABLE IF NOT EXISTS schema_migrations (version INTEGER PRIMARY KEY, updated_at TIMESTAMPTZ ...); current version = MAX(version).storages/engines/rqlite/src/SchemaMigrations.js — JSON row in the existing keyValue table under key migrations/version.storages/engines/mongodb/src/Versions.js, the entire storages/engines/mongodb/src/migrations/ directory (1.9.0.js..1.9.4.js, MigrationContext.js, index.js), storages/engines/mongodb/test/migrations/ (old test fixtures), storages/engines/postgresql/src/VersionsPG.js, storages/interfaces/baseStorage/Versions.{js,d.ts}, storages/interfaces/baseStorage/conformance/Versions.test.js.versions table DDL from DatabasePG.js (it was unused and never populated anyway — _internals.migrations was never registered).migrations / MigrationContext / softwareVersion from both engine _internals.js and the barrel’s registerInternals().storages/engines/{mongodb,postgresql}/manifest.json) to drop the three dead requiredInternals.StorageLayer no longer carries a versions field; components/test-helpers/src/dependencies.js + data.js + databaseFixture.js no longer reference it.dev-migrate-v1-v2 → bin/backup.js --restore operation. No code path in v2 reads pre-v1.9.3 shapes.bin/master.js — replaced storageLayer.versions.migrateIfNeeded() with createMigrationRunner().runAll() gated by migrations.autoRunOnStart (default true). Renamed from cluster.runMigrations in config/default-config.yml.bin/migrate.js (new) — standalone CLI: status / up [--target N] [--dry-run]. Opens storages barrel directly; no HTTP; works whether master is running or not.index.js now exports getMigrationsCapability() returning { id, migrationsDir, getVersion, setVersion, buildContext } or null when the engine is inactive. The runner auto-discovers capabilities across all loaded engines.components/api-server/test/migrations-runner-seq.test.js — [MIGRUN] suite (9 cases): fresh state, single migration, ordered multi-migration, dry-run, target version, idempotent re-run, engine-switch independence (two in-memory engines), failure-stops-run, live-barrel wiring.storages/interfaces/baseStorage/conformance/Versions.test.js removed.storage 13/13, business 126/126, api-server 908/908 (+9 new [MIGRUN] cases).components/api-server/src/routes/reg/records.js: added DELETE /reg/records/:subdomain. Path refactored to share auth + IPC-nudge helpers with the existing POST handler.bin/dns-records.js (new): standalone Node CLI — subcommands list, load <file>, delete <subdomain>, export [file]. Reads/writes PlatformDB directly via storages barrel (no HTTP dependency). Uses js-yaml (already transitively present; no new dependency). Parses --help before boiler init to avoid boiler’s yargs swallowing it.components/api-server/test/dns-records-cli-seq.test.js (new, [DNSCLI]): spawns the CLI as a subprocess and round-trips list / load (including --dry-run and --replace) / export / delete, plus error paths (missing subdomain, malformed file).components/api-server/test/reg-records-seq.test.js: [RR10]-[RR12] cover the DELETE route happy path, missing-auth rejection, and unknown-subdomain 404.Context: end-to-end persistence of runtime DNS records (PlatformDB interface, rqlite backend, DnsServer load-on-start + 30 s refresh + write-through, POST /reg/records persistence, existing [RGRC] test) was shipped earlier in the 2.0.0-pre line. This change adds the DELETE symmetry and the offline-capable admin CLI — the remaining gap for operating DNS records in production without depending on the HTTP API being healthy.
Green-field installs previously needed separate DNS + ACME + reverse-proxy setup before serving a single HTTPS request; multi-core wildcards required a DNS plugin plus manual cert copies across every node. Opt-in letsEncrypt.* folds all of that into the core: issuance, renewal, cluster-wide distribution, hot-swap on rotation.
components/business/src/acme/ (8 files)AtRestEncryption.js — HKDF-SHA256 key derivation + AES-256-GCM envelope. Source-agnostic (caller supplies the byte-string + purpose label). 22 [ATRENC] tests.AcmeClient.js — stateless wrapper over acme-client@5.4.0: createAccount() + issueCert(). Parses leaf-cert validity from the returned PEM so callers get {certPem, chainPem, keyPem, issuedAt, expiresAt}. Injectable acmeLib for unit tests. 9 [ACMECLIENT] tests.certUtils.js — splitCertChain(pem) (leaf vs. chain), parseValidity(pem) (via node:crypto X509Certificate), hostnameToDirName('*.x.com') → 'wildcard.x.com'. 8 [CERTUTILS] tests.CertRenewer.js — glues AcmeClient + AtRestEncryption + PlatformDB. ensureAccount() (idempotent; persists encrypted ACME account), renew({hostname, dnsWriter}) (issues + encrypts + persists + returns metadata), getCertificate(hostname) (decrypted). Strips wildcard prefix in challenge record names. 15 [CERTRENEWER] tests.PlatformDBDnsWriter — default DNS-01 writer for multi-core with embedded DNS. setDnsRecord appends to existing TXT values so apex + wildcard challenges coexist; propagation wait default 15 s.FileMaterializer.js — per-core polling loop. SHA-256 fingerprint-based change detection; atomic disk writes; onRotate fire-and-log semantics. runRotateScript spawns an operator-supplied absolute-path script with PRYV_CERT_* env vars; SIGKILL-timeouts at 30 s with exitCode 124. 11 [FILEMAT] tests.deriveHostnames.js — topology → {commonName, altNames, challenge} in priority order: dnsLess.publicUrl → HTTP-01 single host · core.url → HTTP-01 single host · dns.domain → DNS-01 wildcard + apex. Throws on missing with actionable error. Treats REPLACE ME placeholder as unset. 8 [DERIVEHOSTS] tests.AcmeOrchestrator.js — intervals and start/stop. Materialize tick (every core, default 60 s) + renew tick (only on the CA-holder core with letsEncrypt.certRenewer: true, default 24 h). Both prime on start(). build({config, platformDB, atRestKey, onRotate}) is the operator-facing factory that bin/master.js calls. 10 [ACMEORCH] tests.storages/interfaces/platformStorage/PlatformDB.{js,d.ts} — six new methods: setAcmeAccount/getAcmeAccount (singleton) + setCertificate/getCertificate/listCertificates/deleteCertificate (per hostname, wildcard keys stored as literals e.g. tls-cert/*.mc.example.com).storages/engines/rqlite/src/DBrqlite.js — impl. Keys: tls-acme-account, tls-cert/<hostname>.components/platform/test/conformance/PlatformDB.test.js — 9 new cases (setAcmeAccount / getAcmeAccount singleton + overwrite; setCertificate / getCertificate round-trip + null + overwrite + wildcard keys; listCertificates metadata-only contract; deleteCertificate; namespace isolation between tls-cert / dns-record / user-unique).config/default-config.yml — new letsEncrypt: block (9 keys: enabled, email, atRestKey, renewBeforeDays, staging, tlsDir, certRenewer, onRotateScript, directoryUrl). atRestKey is operator-sync responsibility (same shape as auth.adminAccessKey).bin/master.js — when letsEncrypt.enabled, decode atRestKey (base64 → 32 bytes), call buildAcmeOrchestrator(), start it. Shutdown path (SIGTERM/SIGINT) calls .stop(). Misconfig logs but doesn’t take down master. onRotate callback broadcasts acme:rotate cluster IPC to every live worker.components/api-server/src/server.js — keeps this.httpsServer reference; new reloadTls() re-reads http.ssl.{key,cert,ca}File from disk and calls setSecureContext(). Extracted buildHttpsOptions() helper.components/api-server/bin/server — process.on('message', {type:'acme:rotate'}) → server.reloadTls(). No-op on non-HTTPS workers (hfs, previews, http-only api).components/api-server/src/routes/system.js — new GET /system/admin/certs returning listCertificates() metadata + daysUntilExpiry. admin-key gated by the existing checkAuth.components/business/test/unit/acme-integration.test.js [ACMEINT] — wires real rqlite + mocked acme-client + real FileMaterializer. Three assertions: initial issuance (encrypted in rqlite, decrypted keyPem on disk 0600), no-op on not-yet-due, rotation on forced near-expiry. Raw rqlite row scanned for BEGIN PRIVATE KEY marker — guards against plaintext regressions. Skips gracefully when rqlite isn’t reachable.A 3-level spike against Let’s Encrypt STAGING in _plans/35-letsencrypt-integration-atwork/spike/ proved the end-to-end flow: our dns2 authoritative server published _acme-challenge.test-dns.datasafe.dev TXT records through the full . → .dev → datasafe.dev (Infomaniak) → test-dns (us) delegation chain. LE issued a real staging wildcard cert (*.test-dns.datasafe.dev + test-dns.datasafe.dev). 15 distinct validator IPs across 5+ AWS regions (Frankfurt, Singapore, Stockholm, Oregon, Ohio) all retrieved TXT + CAA correctly — multi-perspective validation fully exercised. Spike also confirmed https.Server.setSecureContext() hot-swaps the cert for new TLS connections without breaking in-flight keep-alive HTTP sessions.
Single-to-multi-core upgrade no longer requires hand-editing override YAML on the new host or copying platform secrets across by hand. An operator runs one CLI on the existing core, transfers a sealed bundle to the new core, and starts the new core in --bootstrap mode. Raft traffic between cores is mutually-authenticated TLS by default.
storages/engines/rqlite/src/rqliteProcess.js buildArgs() — new tls: { caFile, certFile, keyFile, verifyClient, verifyServerName } block translates to rqlited flags -node-ca-cert, -node-cert, -node-key, -node-verify-client, -node-verify-server-name (rqlited 8.x naming).tls: null (default) → zero TLS flags emitted → identical pre-upgrade behaviour. No regression risk for single-core or VPN-protected multi-core.[RQARGS] 14 unit tests cover flag formation across single/multi-core, with/without TLS, verifyClient bool, verifyServerName override.[RQMTLS] integration test spins up two rqlited processes wired with the same self-signed CA + node certs and asserts the cluster forms + a write replicates within 3 s.components/business/src/bootstrap/ (new, 8 modules)ClusterCA.js — ensure() / getCACertPem() / issueNodeCert({ coreId, ip, hostname }). Shells out to openssl (system dep) for X.509 signing; Node’s built-in crypto can generate keys but not sign certs. CA private key never leaves dir (default /etc/pryv/ca, mode 0600); per-issuance temp dir for CSR + node key. EC P-256 keypairs throughout (10y CA / 1y node). 15 [CLUSTERCA] tests.Bundle.js — assemble(input) produces the canonical bundle object (version 1); validate(bundle) rejects unknown versions, missing fields, malformed PEM. Pure (no I/O). Shape: { version, issuedAt, cluster: { domain, ackUrl, joinToken, ca }, node: { id, ip, hosting, url, certPem, keyPem }, platformSecrets: { auth: { adminAccessKey, filesReadTokenSecret } }, rqlite: { raftPort, httpPort } }. 19 [BUNDLE] tests.BundleEncryption.js — encrypt/decrypt using AES-256-GCM keyed off scrypt(passphrase, salt). 16-byte salt, 12-byte nonce, 16-byte tag, base64 + ASCII armor (-----BEGIN PRYV BOOTSTRAP BUNDLE-----). Deliberately uses node’s built-in crypto rather than adding age-encryption — the bundle is only ever consumed by bin/master.js --bootstrap, never manually inspected, and every dep adds supply-chain surface. generatePassphrase() returns 128-bit base64url chunked AbCd-EfGh-IjKl-MnOp for operator readability. 22 [BUNDLEENC] tests.TokenStore.js — file-based one-time join-token lifecycle on the issuing core. Sha256-hashed at rest ({ "<sha256>": { coreId, issuedAt, expiresAt, consumedAt, consumerIp } }); raw token returned only at mint time. Atomic write (tmp + rename) at mode 0600. mint / verify / consume / listActive / revokeByCoreId / purge. Deliberately NOT in PlatformDB — the token consumer is the same core that issued it (the ack endpoint), so cross-core replication is not needed and we avoid adding methods to the PlatformDB interface + two-engine conformance. 26 [TOKENSTORE] tests.DnsRegistration.js — registerNewCore({ platformDB, coreId, ip, url, hosting }) calls PlatformDB’s existing setCoreInfo (with available:false) + setDnsRecord(coreId, { a:[ip] }) + read-merge-write to append ip to lsc.{domain} (the persistent-DNS API is last-writer-wins per subdomain; we want append). unregisterNewCore is the symmetric undo, scoped so it only touches state belonging to this coreId+ip. Two concurrent bootstrap runs could race on lsc; the CLI surfaces a warning that adding cores is a single-operator action. 19 [DNSREG] tests.cliOps.js — orchestrates newCore / listTokens / revokeToken for bin/bootstrap.js. Pure: takes platformDB, caDir, tokensPath, secrets, rqlite ports, output path. Owns the rollback (revoke token + unregister core) on any failure after PlatformDB writes. 7 [BOOTSTRAPCLI] tests with a fake PlatformDB and tmp dirs.applyBundle.js (consumer side) — decrypts + validates a bundle, writes /etc/pryv/tls/{ca.crt, node.crt, node.key} with correct modes (key 0600), generates override-config.yml mapping the bundle into core.{id,url,ip} / dns.domain / dnsLess.isActive:false / auth.{adminAccessKey,filesReadTokenSecret} / storages.engines.rqlite.{raftPort,url,tls.{caFile,certFile,keyFile,verifyClient:true}}. dns + dnsLess are emitted only when bundle.cluster.domain is non-empty (DNSless variant skips both). Override file is mode 0600 (carries admin key). Exposes sha256Fingerprint(certPem) matching openssl x509 -fingerprint -sha256 output. 6 [APPLYBUNDLE] tests.consumer.js (consumer-side driver) — reads bundle from disk, resolves passphrase (passphrase direct arg or passphraseFile with newline-stripping), calls applyBundle, POSTs ack to bundle.cluster.ackUrl with TLS pinned to the bundled CA (https.request({ ca, rejectUnauthorized: true })), deletes bundle on 200, throws on non-200 (bundle is kept so the operator can investigate). httpClient injectable for tests. 7 [BOOTSTRAPCONSUMER] tests.ackHandler.js — makeHandler({ tokenStore, platformDB }) returns a pure req → { statusCode, body } function. 200 ok flips available:true and returns a cluster snapshot; 400 missing field; 401 token unknown / expired / already-consumed / coreId-mismatch (single status code, reasons differentiated in body but no oracle for guessing); 404 token verifies but no pre-registration row. Token is consumed even on the 404 path so the operator must mint a new one. 9 [ACKHANDLER] tests.bin/bootstrap.js (new) — argparse + boiler init + cliOps.newCore / listTokens / revokeToken. Pulls core.url / dnsLess.publicUrl for the ack URL base, auth.adminAccessKey + auth.filesReadTokenSecret for platform secrets (refuses to ship a bundle if either is still on the REPLACE ME placeholder), dns.domain, rqlite raftPort + http port out of storages.engines.rqlite.url.bin/master.js — bootstrap mode runs before @pryv/boiler.init so the override-config.yml it writes lands at the highest precedence in boiler’s load order (override-config.yml → env → argv → ${NODE_ENV}-config.yml → extras). Workers (cluster.fork()) skip the bootstrap block entirely.components/api-server/src/routes/system.js — POST /system/admin/cores/ack route added. checkAuth short-circuits for this single path so the new core can authenticate via the join token instead of the admin key.config/default-config.yml — adds cluster.ca.path (default /etc/pryv/ca) and cluster.tokens.path (default /var/lib/pryv/bootstrap-tokens.json) under the existing cluster: block. Both are PER-CORE — only the issuing core uses them.components/business/src/bootstrap/index.js — barrel exporting all 8 modules.components/business/test/unit/bootstrap-e2e.test.js [BOOTSTRAPE2E] (5 tests) — round-trips cliOps.newCore → consumer.consume → ack route → PlatformDB state with a real http.createServer mounting the ack handler and an in-memory PlatformDB shared between issuer and ack endpoint. Cases: happy path (available flips, bundle deleted, token burned), replay (stashed copy fails 401 already-consumed), wrong passphrase (consumer fails before ack POST attempt, token remains active, pre-registration unchanged), expired token (401 expired), revoke-token after issue (401 unknown, pre-registration unwound). Multi-process / real-rqlited e2e (the reg-2core-seq.test.js pattern) is deferred — not blocking the v2.0.0-pre publication.storages/engines/rqlite/test/.Dockerfile: rqlite binary now bundled in the Docker image (/app/var-pryv/rqlite-bin/rqlited). master.js spawns rqlited directly — previous images lacked the binary. Also removed --omit=optional so sharp installs for the previews worker.bin/master.js: new rqlite.external mode — when storages.engines.rqlite.external: true, master.js skips spawning rqlited and connects to an already-running external instance (multi-core deployments sharing one rqlited on the host).storages/engines/rqlite/src/rqliteProcess.js: new waitForExternal(url, timeoutMs, log) helper.components/api-server/src/methods/mfa.js: removed redundant internal docstring header.just clean-test-data resets MongoDB + rqlitejustfile clean-test-data recipe updated to drop pryv-node-test MongoDB database and wipe the rqlite keyValue table, in addition to the SQLite user index + per-user directories it already cleaned.clean-test-data was still cleaning the obsolete legacy var-pryv/users/platform-wide.db SQLite file. As a result, full-suite runs on a previously-used workstation inherited stale user-* entries from rqlite and orphaned account-field rows from MongoDB, which caused the root-seq.test.js [UA7B] beforeEach integrity check to fail non-deterministically.just clean-test-data && just test all → 1568 / 0 with integrity checks ENABLED. Same for just test-pg all → 1543 / 0. No DISABLE_INTEGRITY_CHECK=1 workaround needed on sequential runs anymore. Parallel runs still use the workaround because parallel workers share state across processes.storages/interfaces/backup/FilesystemBackupWriter.js writeChunkedJsonlFiles() — compressed-mode chunking check now also fires when rawSize >= maxChunkSize, not only every 100 items. Small datasets (< 100 items) with aggressive maxChunkSize previously produced a single chunk regardless of target; they now respect the soft limit. Large datasets are unaffected (100-item batch check still dominates; the raw-bytes trigger is a lower bound).components/business/test/unit/backup/filesystem-writer-reader.test.js were subtly wrong — they used highly compressible payloads ('Hello world '.repeat(5), short fixed strings) that gzip to almost nothing, so the compressed size never reached maxChunkSize. Updated to use non-compressible pseudo-random payloads so the chunking assertions are deterministic.'round-trips a single event larger than maxChunkSize (soft-limit semantics)' documents that an individual oversized item is written to exactly one chunk — chunks cannot split items.config/plugins/core-identity.js now honors an explicit core.url YAML override as the highest-priority source for core:url. Falls back to id+domain derivation, then to dnsLess.publicUrl.Platform.js:
#coreUrlCache: Map<coreId, url> populated by _refreshCoreUrlCache() from PlatformDB on init() and on registerSelf(). Lets coreIdToUrl() stay synchronous (~10 call sites in api-server) while honoring explicit URLs registered by other cores.registerSelf() now writes url: this.coreUrl || null into the core info row so DNSless multi-core deployments can advertise their externally-correct URL.coreIdToUrl(coreId): cache lookup → derivation from id+domain → self URL fallback. NOTE: cache stays cold for changes made by OTHER cores until the next init() — periodic refresh for dynamic cluster membership is a planned follow-up.components/middleware/src/checkUserCore.js — wrong-core check on /:username/*. Returns HTTP 421 Misdirected Request with { error: { id: 'wrong-core', message, coreUrl } } when platform.getUserCore(username) !== platform.coreId. No-op in single-core mode and for unknown users (existing 401/404 paths handle them). Test hook _resetPlatformCache() exposed for cross-test isolation.components/middleware/src/index.js exports the new middleware as middleware.checkUserCore.components/api-server/src/routes/root.js mounts middleware.checkUserCore on Paths.UserRoot + '/*' BEFORE getAuth and initContextMiddleware so wrong-core requests don’t pay the cost of user/access loading.components/api-server/test/reg-multicore-seq.test.js: 5 wrong-core middleware tests [MC09A..MC09E] (wrong core, right core, unknown user, /reg bypass, single-core no-op) and 3 explicit-URL tests [MC10A..MC10C] (cache hit, derivation fallback, end-to-end through middleware).PlatformDB interface gains setDnsRecord(subdomain, records) / getDnsRecord(subdomain) / getAllDnsRecords() / deleteDnsRecord(subdomain). The rqlite backend (storages/engines/rqlite/src/DBrqlite.js) implements them on the existing keyValue table using a dns-record/{subdomain} key prefix — no schema migration needed.Platform.js gains delegating methods for the four DNS record operations.DnsServer (components/dns-server/src/DnsServer.js) now loads runtime DNS records from PlatformDB on start() and refreshes them every 30s by default (platformRefreshIntervalMs constructor option). YAML dns.staticEntries are authoritative — admin runtime entries cannot shadow them. updateStaticEntry() is now async and persists to PlatformDB before updating the in-memory map. New deleteStaticEntry() mirror.POST /reg/records (components/api-server/src/routes/reg/records.js) writes to PlatformDB first, then sends an IPC nudge to master so the local DnsServer refreshes immediately. Other cores in a multi-core deployment pick up the change via the periodic refresh.bin/master.js IPC handler refreshes from PlatformDB on dns:updateRecords instead of trusting the IPC payload — single source of truth.default-config.yml annotated with # === PER-CORE / PLATFORM-WIDE / BOOTSTRAP / MIXED === section headers for every block.README.md explaining the three categories and how multi-core operators must respect the split.Platform.registerSelf() logs a [platform-config-snapshot] line on boot with this core’s observed values for known platform-wide keys (dns.domain, integrity.algorithm, versioning.deletionMode, uploads.maxSizeMb) plus a SHA-256 hash of auth.adminAccessKey. Operators can compare these across core logs to detect drift without the admin key value ever appearing in logs. auth.adminAccessKey is confirmed YAML-only (BOOTSTRAP, secret, never moves to PlatformDB).platform_config table with live drift warnings and per-key migrations is a planned post-v2 follow-up.components/business/src/mfa/Profile.js — per-user MFA state model (content + recovery codes); replaces lodash _.isEmpty with native checkService.js — abstract base for SMS providers. Takes a plain mfaConfig object (not boiler) for DI-friendly tests. Static replaceAll/replaceRecursively helpers (immutable — original mutated input).ChallengeVerifyService.js — two-endpoint SMS provider (external SMS service generates + validates the code)SingleService.js — single-endpoint SMS provider (service-core generates the code + validates locally, templates it into an HTTP call that only delivers the SMS)generateCode.js — drops bluebird, uses node:util.promisify + node:crypto.randomBytesSessionStore.js — in-memory Map<mfaToken, {profile, context, _timeout}> with TTL via per-session setTimeout().unref(). Single-core only; multi-core sharing deferred.index.js — barrel with createMFAService(mfaConfig) factory and getMFAService/getMFASessionStore process-wide singleton accessorscomponents/api-server/src/methods/mfa.jsmfa.activate, mfa.confirm, mfa.challenge, mfa.verify, mfa.deactivate, mfa.recover on the v2 API. Added to components/audit/src/ApiMethods.js ALL_METHODS (required for registration). mfa.recover is in WITHOUT_USER_METHODS.services.mfa config per-invocation (not module load) so tests can inject config dynamically.errors.factory.apiUnavailable (HTTP 503) when MFA is disabled server-wide.saveMFAProfile uses the Profile storage’s dot-notation converter shape: {data: {mfa: X}} for set and {data: {mfa: null}} for unset — the converter turns NULL leaves into $unset['data.mfa'].components/api-server/src/routes/mfa.jsPOST /:username/mfa/* endpointsactivate and deactivate use loadAccessMiddleware (personal access token required); confirm/challenge/verify extract mfaToken from the Authorization header (supports raw token and Bearer <token> shapes) and pass it via params.mfaToken; recover is unauthenticated.Paths.MFA = /:username/mfa entrycomponents/api-server/src/methods/auth/login.jsmfaCheckIfActive step appended to the auth.login method chain. When MFA is enabled server-wide AND the user has profile.private.data.mfa set, it calls mfaService.challenge(), stashes the issued {user, token, apiEndpoint} in the SessionStore under a fresh mfaToken, and replaces the response with {mfaToken} — the caller must then call mfa.verify to release the real token.profile.mfa → step is a no-op (login response unchanged).config/default-config.ymlservices.mfa block with mode: disabled default. SMS endpoints are empty strings; sessions.ttlSeconds: 1800. Existing deployments are fully backwards-compatible.components/business/test/unit/mfa/ — 23 unit tests across generateCode (2), Profile (3), Service (6), createMFAService factory (5), SessionStore (7)components/api-server/test/mfa-seq.test.js — 15 acceptance tests ([MFAA]/[MA*]) covering the full activate→confirm→login→challenge→verify→deactivate→recover lifecycle with nock-mocked SMS endpoints and nock.disableNetConnect() for fast failure on missing mocksmfa to the methods list in components/test-helpers/src/helpers-base.js AND helpers-c.js (the latter hardcodes the list)require('api-server/src/methods/mfa') to components/api-server/test/helpers/core-process.js (multi-core tests)service-mfa’s separate HTTP proxy, Dockerfile, runit lane. Repo is archived (final commit adds README pointer to the merge commit).service-mfa/src/business/pryv/Connection.js (replaced by direct userProfileStorage + usersRepository calls), its own middlewares/, its own errorsHandling.js (replaced by the core errors.factory).storages.platform.engine default flipped from sqlite → rqlite. rqlite is now the only supported runtime platform engine in v2.platformStorage: removed from storages/engines/sqlite/manifest.json, dropped createPlatformDB export from storages/engines/sqlite/src/index.js, deleted DBsqlite.js and the [SQPF] SQLite PlatformDB conformance test. SQLite remains in use for baseStorage, dataStore, and auditStorage.mongodb and postgresql engines still ship PlatformDB implementations for conformance tests, but cannot be selected as the runtime platform engine via config.bin/master.js always spawns and supervises an embedded rqlited (no engine guard, no external flag check). The storages.engines.rqlite.external config (previously available) has been removed — master.js owns the rqlited lifecycle in both single- and multi-core mode.lsc.{dns.domain} to join peers.bin/migrate-platform-to-rqlite.js moved to dev-migrate-v1-v2/migrate-platform-to-rqlite.js. It is no longer needed for in-v2 single→multi-core upgrades (no migration step at all). Retained in the v1→v2 toolkit for the same shape of work, with a header note explaining the rework needed before reuse.storages/engines/rqlite/scripts/setup — downloads rqlited v9.4.5 from GitHub releases (Linux/macOS, amd64/arm64), idempotent, mirrors mongodb patternstorages/engines/rqlite/scripts/start — single-node foreground/background launcher with pidfile and /readyz waitstorages/engines/rqlite/manifest.json: declared scripts.setup and scripts.startscripts/setup-dev-env: invokes rqlite setup after mongodbINSTALL.md: rqlite added to prerequisites; minimal config updated; data/rqlite-data/ documentedSINGLE-TO-MULTIPLE.md: rewritten — removed manual rqlite install + data migration steps; new flow is DNS → config → restart → deploy second core (224 → 145 lines)README.md: storage engines table updated (rqlite added, platform removed from MongoDB/PostgreSQL/SQLite rows)README-DBs.md: rewrote “Platform Wide Shared Storage” section to describe the rqlite-everywhere modelstorages/pluginLoader.js: stale platform: engine: sqlite example updatedRestoreOrchestrator._restorePlatform: v1 backups (and any future raw exports) write platform entries as {key, value} straight from the legacy SQLite/MongoDB platform-wide store, but platformDB.importAll expects the parsed shape {username, field, value, isUnique}. The orchestrator now bridges both shapes via a new parseRawPlatformEntry helper, so v1→v2 migrations restore platform data correctly. v2→v2 round-trips still pass entries through unchanged.pm@perki.com → perki.storages.engines.rqlite.external config: skip embedded rqlited, connect to external instancepublic-url.js plugin: generate api/register/access service info for multi-core modecore-identity runs before public-urlsubdomainToPath middleware: skip core’s own subdomain in multi-core mode/reg/cores: check shared PlatformDB before local users_index for cross-core lookupsINSTALL.md: standalone HTTPS (backloop.dev / custom certs) + nginx reverse proxy guideSINGLE-TO-MULTIPLE.md: step-by-step single→multi-core upgrade guidebin/migrate-platform-to-rqlite.js: reads users from base storage, populates rqlite platform DB--ignore-scripts + rebuild native modules, audit syslog active:false supportstreamId/profileId → idFilesystemBackupWriter maxChunkSize now applies to the compressed output size (was uncompressed)bin/backup.js accepts --target-file-size <MB> as alias for --max-chunk-sizestorages/interfaces/backup/)BackupWriter/BackupReader interfaces with filesystem implementation_id/__v/userId, promotes _id to id (except streams)BackupOrchestrator: snapshot consistency (snapshotBefore), --incremental mode (auto-detects per-user timestamps)RestoreOrchestrator: conflict detection, --skip-conflicts, --overwrite, --verify-integrity with rollbackexportDatabase()/importDatabase()--incremental to write to existing path)components/business/src/integrity/IntegrityCheck.js)--verify-integrity)bin/backup.js: full backup/restore CLI (all-users, single-user, incremental, compressed)bin/integrity-check.js: standalone per-user integrity verification (--user, --json, exit code 0/1)exportAll, importAll, clearAll to UserStreams and UserEvents interfacestools/coverage/)NODE_V8_COVERAGE + c8 report (replaces NYC)collect.js: bypasses components-run, runs mocha from project root via node directlypg-early-init.js: fixes barrel init race — injects PG config before global.test.js locks to MongoDBrun.sh: orchestrates 3-engine coverage (MongoDB + PG + SQLite), merged HTML reportDynamicInstanceManager (random port), fixed cache cleanup config key, async getFiles(), test assertion fix — 15/0 (was process crash)activateWebhook() for PG modeencryption.js — engines now use require('utils').encryptionfindStreamed/findDeletionsStreamed stubs (MongoDB + PG), stateToDB, LocalTransaction.commit/rollback, Database.findStreamed, hasStreamPermissions, User.getEvents/getUniqueFields, MallUserEvents.getStreamed, storage.getDatabase/getDatabasePG, pluginLoader.getConfigForDISABLE_INTEGRITY_CHECK removed from test recipesevent_streams(user_id, stream_id, event_id) for stream-parent queriesDatabasePG.ensureConnect() with promise guard — fixes pg_type_typname_nsp_index race condition when multiple callers initialize concurrently_initSchemaOnce() with _schemaReady flagconnected flag set only after schema is ready (prevents queries against missing tables)storages.engines.postgresql.auditPoolSize — configurable audit pool size (default 5)justfile: removed DISABLE_INTEGRITY_CHECK=1 from test-pg and test-pg-parallel recipestools/performance/)--sweep 1,5,10,25,50 produces comparison tablesperf-clean, perf-seed, perf-run, perf-fullbin/compare.js for side-by-side result analysisconfig/ (merged from per-component configs)storages.{base,platform,series,file,audit}.engine + storages.engines.<name>setUserUniqueFieldIfNotExists() atomic method (all 4 backends)storages/engines/rqlite/) for distributed PlatformDBvalidateOnPlatform → createUser → buildResponsePlatform.js: removed service-register HTTP client, added validateRegistration() with invitation tokens, reserved words, atomic unique field reservationinvitationTokens seeds on first boot; tokens consumed on registrationservice_register.js, reserved-words.json (124K words) copied to platform componentrepository.js: renamed updateUserAndForward → updateUser, removed skipFowardToRegister parametertestsSkipForwardToRegister config key, isDnsLess conditionals from registration logicisDnsLess guard)core.id → FQDN, self-registration in PlatformDB-http-adv-addr)lsc.{domain}components/dns-server/ — dns2-based DNS server for {username}.{domain} resolutionPOST /reg/records admin endpoint for runtime DNS entry updatesroutes/reg/legacy.js — backward-compatible service-register endpointsreg-multicore-seq.test.js)reg-2core-seq.test.js) — real rqlite + 2 child processes + DNSdns-server.test.js) — dns.promises.Resolver + raw dgramreg-gap-features-seq.test.js)reg-legacy-seq.test.js)core-process.js — child process boot script for integration testsgm (GraphicsMagick wrapper, requires system binary) with sharp (npm-native, bundles libvips)apt-get install graphicsmagick from Dockerfile — no system image dependencies for previewsmaster.js — previews worker always starts when enabledbluebird usage from event-previews.js (sharp is Promise-native)components/externals/ — runs lib-js test suite (169 tests) via just test externalspublic-url.js: preserve service.assets and service.features from config in dnsLess modelibjs-test-config.yml for dnsLess + HTTPS + pryv.me assets configurationexternal-ressources/ from eslint and source-licensermetadata and tprpc components, tchannel/protobufjs dependenciesbuild/webhooks/ (Dockerfile + runit)bin/master.js — single master process using Node.js cluster modulecluster:apiWorkers, default 2)cluster:hfsWorkers, default 1, 0 = disabled)PRYV_BOILER_SUFFIX (-wN, -hfsN)cluster:previewsWorker, default true)node bin/master.js — replaces runit-based orchestrationcluster:runMigrations, default true)build/core/, build/hfs/, build/preview/ (Dockerfiles + runit scripts), Dockerfile.component-intermediate, Dockerfile.common-intermediateopenSource:isActive flagopenSource:isActive config flag and all gated code — features always enabled: webhooks, HFS/series events, distributed cache sync, email check routeisOpenSource fields and constructor logic from Application, Server, Manager, NamespaceContext classesopenSourceSettings config reads and conditional series creation/integrity/notification logic in events.jsisSynchroActive from !config.get('openSource:isActive') to true in cacheopenSource: config sections from 5 config files (default, development, test, hfs-server, build/test)openSource (12 test files)www/register package requires in application.js that would crash if ever reachedUserAccountStorage with account field CRUD methods (getAccountFields, setAccountField, deleteAccountField, getAccountFieldHistory)accountStore adapter implementing pryv-datastore interface, wrapping baseStorage account operationsaccountStore in Mall alongside local + audit storesstoreDataUtils.js routing: :_system:/:system: prefixes → account store (was local)forbidSystemStreamsActions() from streams.js — account store handles rejectionfilterNonePermissionsOnSystemStreams() from utility.js — standard permissions applyForbiddenAccountStreamsModification error constantconsole.log('XXXXX') traps from User.jsisAccountStreamId hard block from AccessLogic — includedInStarPermissions handles itnone prepend from serializer iterator to single STREAM_ID_ACCOUNT constantsystemStreamFilters.js in test-helpers for test-only prefix helpersinit() calls from hfs-server tests:_system:helpers stream (parent of active/unique markers)structuredClone() to prevent readableTree mutationbusiness/src/system-streams/features.js, then inlined as plain stringsSystemStreamsSerializer → accountStreams, forbiddenStreamIds → hiddenStreamIds, removePrefixFromStreamId → toFieldName, addCorrectPrefixToAccountStreamId → toStreamId, indexedFieldsWithoutPrefix → indexedFieldNames, uniqueFieldsWithoutPrefix → uniqueFieldNamesserializer.js — content moved to system-streams/index.jsnats/NATS variable names, function names, comments, and config references to generic transport/Transport termsNATS_MODE_ALL/KEY/NONE → TRANSPORT_MODE_ALL/KEY/NONE (backward compat aliases kept)initNats() → initTransport(), isNatsEnabled() → isTransportEnabled(), setTestNatsDeliverHook() → setTestDeliverHook()NATS_CONNECTION_URI in webhooks, axonMessaging export alias, nats:uri compat fallbacknats npm package with zero-dependency TCP pub/sub broker using Node.js net moduletcp_pubsub.js: embedded broker/client — first process binds port, others connect as clientsnats:uri → tcpBroker:portnats-server/ binary directory and scripts/setup-nats-servernats npm dependencytest_messaging.js (IPC-based EventEmitter + process.send()), deleted axon_messaging.jsaxon-* message names to test-* across all test files (~20 files)axonMsgs/axonSocket variables to testMsgs/testNotifieraxon npm dependency and axonMessaging config sectionsferretDB/ directory, test-ferret justfile recipe, isFerret config/property, FerretDB connection string, ferretIndexAndOptionsAdaptationsIfNeeded(), FerretDB duplicate error handling, FerretDB test guardsDatabase.isDuplicateError(): FerretDB branch had missing return, causing all errors to be reported as duplicatesproduceInfluxConnection() with async produceSeriesConnection() factory in hfs-server and api-server test helpersgetTimeDelta() helper to normalize time values across InfluxDB (INanoDate) and PG (numeric)produceMongoConnection → produceStorageConnection globally across 18 test filesstore_data.test.js, batch.test.js, deletion-seq.test.js to use engine-agnostic series queries/assertionspg_connection.js query(): changed delta_time * 1000 → delta_time / 1e6 (delta_time stores nanoseconds via InfluxDateType.coerce, not seconds)time placed before JSONB fields to match InfluxDB column orderexportDatabase() and importDatabase() for consistent nanosecond↔millisecond conversiondeletion.js deleteHFData() now dispatches to PG or InfluxDB based on storage engine (was hardcoded to InfluxDB)MOCHA_PARALLEL=1) from integrity check disable (DISABLE_INTEGRITY_CHECK=1) in helpers-base.js and helpers-c.js — fixes cache tests failing under just test-pgWebhook.test.js user object to include id property — PG requires non-NULL user_id for SQL equality comparisons (MongoDB was tolerant of undefined via collection naming)FollowedSlices.js, FollowedSlicesPG.js, followedSlices.js (methods), followed-slices.js (routes), schema files, test filedependencies.js — always reconfigures via StorageLayer (removed STORAGE_ENGINE guard)parallelTestHelper.js — always uses storage.getStorageLayer() instead of engine switchtest-helpers.js — produceConnection() always returns StorageLayerprofile-personal.test.js — uses storageLayer.profile instead of engine switch// CLAUDE: marker commentsapplication.js — removed unconditional storage.getDatabase() callcontext.js — removed mongoConn property; constructor no longer requires database connectionmetadata_cache.js — MetadataLoader.init() no longer takes databaseConn param (was unused)integrity-final-check.js — early return for non-MongoDB engines (uses raw MongoDB cursors)SessionsPG — callback-based sessions with JSONB containment for getMatchingPasswordResetRequestsPG — callback-based password reset with expiration on readVersionsPG — async versions with migration runnerBaseStoragePG — base class providing full UserStorage interface with SQL query building:
camelCase↔snake_case mapping, JSONB serialization, MongoDB-style query→SQL translation
($gt, $ne, $in, $or, $type, $set, $unset, $inc, $min, $max, JSONB dot-notation)AccessesPG — token generation, integrity hash, integrity-aware delete/updateOneWebhooksPG — aggressive field unsetting on soft-deleteProfilePG — JSONB data with key-value set updatesStreamsPG — tree build/flatten via treeUtils, cache invalidationuserAccountStoragePG — password history, key-value store (StoreKeyValueData)usersLocalIndexPG — username↔userId mappingDBpostgresql — platform unique/indexed fieldslocalDataStorePG — DataStore factory implementing @pryv/datastore interfacelocalUserEventsPG — full events API with junction table event_streams for stream queries,
intermediate query format→SQL conversion, streaming supportlocalUserStreamsPG — streams API with tree building, cache integrationLocalTransactionPG — PG transaction wrapperpg_connection.js — implements InfluxConnection interface using series_data tablestream_ids JSONB column to events table for denormalized readsStorageLayer._initPostgreSQL instantiates all PG backendsAdded storageEngine config key (‘mongodb’ |
‘sqlite’ | ‘postgresql’) to default-config.yml |
database:engine, storageUserAccount:engine, etc.)postgresql connection config block (host, port, database, user, password, max)storage/src/getStorageEngine.js — unified engine resolution with validationstorage/src/DatabasePG.js — connection pooling via pg, schema DDL for all tablesensureConnect(), waitForConnection(), query(), getClient(), withTransaction(), initSchema(), close()isDuplicateError(), handleDuplicateError() (mirrors MongoDB pattern)StorageLayer.js — refactored to dispatch to _initMongoDB/_initSQLite/_initPostgreSQLstorage/src/index.js — engine routing for getUserAccountStorage(), getStorageLayer(); exports DatabasePG, getDatabasePG, getStorageEngineusersLocalIndex.js — engine routing via getStorageEnginemall/src/index.js — datastore selection by engineplatform/src/getPlatformDB.js — PlatformDB selection by enginehfs-server/src/application.js — series connection selection by enginepg (node-postgres) to root dependenciesdropCollection() with removeAll() in business/src/auth/deletion.js (interface compliance)webhooks, acceptance/accesses, login-parallelgetApplication() shared state)permissions-seq.test.js sections AP01, AP02, YE49 → new permissions.test.js (Pattern C, parallel-safe)events-seq.test.js (covered by events-patternc.test.js)streams-seq.test.js (covered by streams-patternc.test.js)events-mutiple-streamIds.test.js for parallel-mode debuggingUserStorage interface with validateUserStorage() in storage/src/interfaces/exportAll, importAll, clearAll) to BaseStoragevalidateSessions() interface, migration methods (exportAll, importAll)validatePasswordResetRequests() interface, migration methods (exportAll, importAll)validateVersions() interface, migration methods (exportAll, importAll)createUserAccountStorage() factory in storage/src/interfaces/_exportAll, _importAll, _clearAllvalidateUsersLocalIndexDB() validation in storage/src/interfaces/exportAll, importAll, clearAll) to Mongo and SQLite classesusersLocalIndex.jsvalidatePlatformDB() validation in platform/src/interfaces/exportAll, importAll, clearAll) to Mongo and SQLite classesgetPlatformDB.jscreateEventFiles() factory + validateEventFiles() in storage/src/interfaces/getEventFiles.jsInfluxConnection interface with validateInfluxConnection() in business/src/interfaces/exportDatabase, importDatabase) to InfluxConnectionbusiness.series.interfaces.InfluxConnectionUserSQLiteStorage interface with validateUserSQLiteStorage() in storage/src/interfaces/UserSQLiteDatabase interface with validateUserSQLiteDatabase() in storage/src/interfaces/exportAllEvents, importAllEvents) to UserDatabasestorage.interfaces.UserSQLiteStorage and storage.interfaces.UserSQLiteDatabasestorage/src/index.js exports all interfaces under interfaces keyswitchSqliteMongo/) simplified using standardized exportAll/importAll/clearAllcomponents/audit/src/Audit.jsfactory.periodsOverlap error and ErrorIds.PeriodsOverlapbackwardCompatibility.systemStreams.prefix config from all config filesisStreamIdPrefixBackwardCompatibilityActive variable and all guarded code in events.js, streams.js, accesses.js, eventsGetUtils.jsbackwardCompatibility.jsChangeStreamIdPrefixStream.jsdisableBackwardCompatibility property and disable-backward-compatibility-prefix header from MethodContext.jsPATTERN_C_BACKWARD_COMPAT env var handling from test helpers/register/create-user endpointPOST /register/create-user route from system.js[ZG1L]passwordHash parameter kept (still used by standard POST /system/create-user)streamId (singular) backward compatibilitystreamId property from event JSON schema (event.js)anyOf (streamId or streamIds) to required: ['type', 'streamIds']normalizeStreamIdAndStreamIds in events.js — removed BOTH_STREAMID_STREAMIDS_ERROR and all event.streamId = event.streamIds[0] assignmentsSetSingleStreamIdStream.js (no longer needed to add streamId to output)SetSingleStreamIdStream pipe from eventsGetUtils.jsevent.streamId assignment from SetFileReadTokenStream.jsbackwardCompatibility.js, AddTagsStream.jsbackwardCompatibility.tags from all config filesevents.js (replaceTagsWithStreamIds, putOldTags, createStreamsForTagsIfNeeded, cleanupEventTags, migrateTagsToStreamQueries)AccessLogic.jstags from event schema and events.get query params1.7.0.jspreviews-server/event-previews.js to use canGetEventsOnStream instead of removed canGetEventsOnStreamAndWithTagspermissions-tags.test.js from test lists/service/infos route duplicatenewSreamIds → newStreamIds in events.jsawait in repository.js