Initially Pryv.io was built on top of MongoDB with separated collection per user. This initial design allowed to isolate peruser data on the file system.
This design has a drawback as MongoDB was consuming a fixed amount of RAM per collection and the with growing sets of users (over 40’000 per node) 16Gb was needed. In v1.6.0 an option to merge the Mongo’s collection was added, resulting in an average RAM requirement of 4Gb for 100’000 users.
From v1.7.0 Sqlite has been investigated in order to provide back the ability to isolate peruser data on the file system. The motivation is to provide full control over the user’s data in order to facilitate and prove the “right to be forgotten.”
From v1.8.0 a Sqlite version for Event has been provided on top of the datastore abstraction.
From v1.9.x FerretDB has been implemented as on optional replacement of MongoDB.
Since v1.9.2 Pryv.io can be deployed in “full-cloud” setup without relying on the file system. This can be done by configuring all storage modules to use MongoDB. For the attachments and S3 implementation is in development.
For future v1.9.3 Pryv.io will be also capable in being “full local” with only SQLite databases.
base code: components/storage/src/userLocalDirectory.js
Localization of user data on the host file system, usually in var-pryv/users then a directory path is constructed using the 3 last characters of the userId and the userId.
Exemple with userId c123456789abc: var-pryv/users/c/b/a/c123456789abc/
In this directory, the attachments and any user attributed data and sqlite db should be stored.
base code: components/storage/src/userLocalIndex.js
This database is a per-server index to map userId and userName. In the future it could be extended to allow user aliases.
var-pryv/user-index.dbid4name and stored in the main host database pryv-nodeSettings to activate MongoDB/ferretDB instead of SQLite: storageUserIndex:engine = 'mongodb'
Script to migrate userIndex from SQLite to MongoDB: read first
LOGS=info node components/storage/src/migrations/switchSqliteMongo/usersIndex.js --config configs/api.yml
base code: components/storage/src/userAccountStorage*.js *: Mongo or Sqlite
This database contains the password and passwords history of the user.
account-1.0.0.sqlite .passwords and stored in the main host database pryv-nodeSettings to activate MongoDB/ferretDB instead of SQLite: storageUserAccount:engine = 'mongodb'
Script to migrate from SQLite to MongoDB: read first
LOGS=info node components/storage/src/migrations/switchSqliteMongo/userAccountStorage.js --config configs/api.yml
base code: components/platform
This database contains all indexed and unique fields for users such as emails and custom systems streams data.
In the Enterprise version of Pryv, it acts as a local cache and report to service-register being the main index. For Open-Pryv.io platformDB should evolve in a shared database between running service-core.
var-pryv/platform-wide.dbSettings to activate MongoDB/ferretDB instead of SQLite:storagePlatform:engine = 'mongodb'
Script to migrate from SQLite to MongoDB: read first
LOGS=info node components/storage/src/migrations/switchSqliteMongo/platformDB.js --config configs/api.yml
base code: components/storage/src/localDataStore and localDataStoreSQLite
Main storage for events , streams & attachments this implementation follows the modular API of datastore abstraction.
base code: components/storage/src/user
Only implemented for MongoDB/FerretDB - Expecting full SQLite implementation in v1.9.3
storage with full-mongo as indexes for password is not yet created. Run just test-full-mongo storage to reproduceMongoDBstorage*:engine setting to mongodb yet !storage*:engine = 'mongodb'