“It looks like a node, swims like a node, quacks like a node… but it only serves data”. Tweet from Santiago Carmuega, development leader.
The name chosen for the development is Dolos, the spirit of cunning deception, in Greek mythology, since it is not what it seems.
Traditionally, nodes assume one of two roles: “relay” or “producer” of blocks. However, there is a third function that needs to be treated separately, and that is performed by nodes whose purpose is to resolve local state queries, or to serve as a data source for tools such as accounting data.
These nodes have different requirements than producer/relay nodes, and have better performance/cost optimizations.
The proposal is to develop a new node, tuned to solve a very limited scope, that is, to maintain an up-to-date copy of the ledger and respond to customer queries, while requiring a small fraction of the network resources compared to the previous ones producer/relay nodes.
In the previous graph you can see the difference between both types of nodes.
By narrowing the scope and selecting a different set of trade-offs, you can optimize this role to have a small resource footprint.
Implementing a “data node” should be relatively inexpensive compared to a full node, which does all the heavy lifting of consensus on the ledger.
Since the main purpose is to serve as a data source, one could also go the extra mile and add more communication protocols to facilitate interoperability. A gRPC endpoint for local state queries, and mini chain synchronization protocols will allow seamless access from Python, Go, Node, etc.
gRPC Remote Procedure Calls, is a modern open source high performance Remote Procedure Call (RPC) framework that can run in any environment.
The data nodes will share some of the features with the main Cardano node:
- Mini node-to-node and node-to-client chain synchronization protocol Mini node-to-node
- block recovery protocol
- Mini node local status query protocol to client
By decoupling the storage mechanism, you can create (nearly) stateless, read-only replicas to support ever-increasing workloads all consuming the same data (for example: S3 Bucket, GCP Bucket, NFS, etc.).
Dolos will be developed as an open source project, using Rust as its main development language. The code will make heavy use of the Pallas developed by this team, which is already available as version 0.11.1.
The workload carried out by this type of node cannot currently be executed on the Cardano node.
This new node type will also provide features not currently available in the main Cardano node, such as HTTP/JSON endpoint for common local state queries, gRPC endpoint for local state queries, and chain synchronization procedure, different storage options, including NFS, S3, and GCP buckets, and low memory consumption (allowed by scope trade-offs).
It will bring the following optimizations:
- drastically limits the amount of memory required to run the node
- switches to storage solutions with different trade-offs (e.g. S3, NFS, etc.)
- provides friendlier alternative connection protocols for data queries data (eg REST, gRPC)
- provides an authentication layer against API endpoints
The goal of this project is to provide a very limited and focused version of the Cardano node that DevOps can use as a cost effective and efficient option to implement data nodes along with producer/relay nodes, useful in the following scenarios:
- As a data source for well-known tools like DB-sync, Ogmios, CARP, Oura, etc.
- As a fast, low-resource node to synchronize other producer/relay nodes.
- As a ledger data source that scales dynamically based on query load.
- As a node that leverages cloud/network storage technology instead of mounted drives.
- As a node that scales out, enabling highly available topologies.
- As a low-resource local node to resolve local state queries.
The Roadmap
Milestone #1: Proof of Concept (PoC)
- 1 month development
- 1 full-time Rust developer
- 1 part-time SRE
- Deliverables
- Working Prototype
- ChainSync client
- Local file-system storage
- Subset of local state queries
Milestone #2: Features
- 2 month development
- 1 full-time Rust developer
- 1 part-time SRE
- Deliverables
- Working Prototype
- More storage options
- Authentication mechanism
- gRPC endpoint
- HTTP/JSON endpoint
- ChainSync / BlockFetch server
Milestone #3: Hardening / Documentation
- 1 month development
- 1 part-time Rust developer
- 1 part-time SRE
- 1 full-time technical writer
- Deliverables
- Fully-Functional v1
- Performance optimizations
- Bug-fixing
- Documentation site
If the project achieves good adoption, the team intends to resubmit a proposal to Catalyst for a version 2.
Project progress will be measured by published progress that matches the reach of predetermined milestones.
Each milestone will be presented as a partial but functional “release” within the repository, and can be tracked through Github, using its project management tool.
Direct communication with the development team will be possible through the TxPipe Discord server, and weekly update summaries will be posted through the TxPipe Twitter account.
There will be no strict and quantifiable measurement until the Proof of Concept for this project, Milestone #1, is developed.
The Budget
Total requested: USD 75,680.
- Project Manager: 80 hours x USD/h 60 = USD 4,800
- Rust Developer: 560 hours x USD/h 75 = USD 42,000
- Site Reliability Engineer: 320 hours x USD/h 70 = USD 22,400
- Technical Writer: 180 hours USD/h 36 = USD 6,480
The Team
TxPipe is a small team of experienced developers, and they present themselves as passionate about Open Source and Cardano.
TxPipe is the team behind existing projects like Oura (1) and Pallas.
They specialize in back-end development, data pipelines, and cloud architecture. Rust is the language of choice.
Santiago Carmuega will lead the software development. He is a senior developer who claims to have over 20 years of software development experience and is very active in the Cardano open source ecosystem.
Github: https://github.com/scarmuega Twitter: https://twitter.com/santicarmuega
Alejandro Drabenche will be the SRE in charge of validating, implementing and testing the project at each milestone. He is a senior system administrator with more than 15 years of experience. He has been working on blockchain for more than 5 years.
Florencia Luna will be in charge of the technical writing. She is a junior developer with a background in technical writing.
Federico S. Weill will be the director of the project. He is a senior project manager, has a PhD in science, and has led more than 10 research projects over the last 20 years managing resources and people.
They plan to hire a new software developer with Rust experience to contribute to the codebase starting at milestone #2.
You can read the original proposal on Catalyst: Dolos: Cardano “Data Node”
. . .
