Polkadot conformance testing
Dec 22, 2023
In this article, we will present our prototyping and research work regarding Polkadot conformance tests. Polkadot exists as a protocol and has different implementations written in different programming languages. Today, most nodes likely run the official Rust implementation. Thus, introducing a new node with a different client could lead to bugs and unwanted behaviours if it is not compliant with the specifications.Addressing this issue is a challenging problem, for which we have attempted various approaches.
Polkadot conformance tests existing work
Web3 Foundation has already funded projects to help testing the node:
Web3 Foundation Polkadot conformance tests
This project relied on Foreign Function Interface (FFI) code to write unit tests for each implementation in various languages. While this approach does not require any modifications on the node part, it is heavy to maintain. It demands deep knowledge of each node's code base, and FFI code is challenging to write, read, and maintain. Development of this testing framework ceased last year.
Fuzzing is a testing technique that loads functions with randomly generated data to find unwanted behaviour. Polkadot is relying on a P2P message interface that can be accessible by anyone, including malicious actors. Testing that random inputs won’t crash a node is a good practice. However it won’t protect against human error in specification implementation or more elaborated attack.
Zombienet is a tool that spawns ephemeral Polkadot/Substrate networks on demand. It will test how an assembly of nodes would work together in a real network. It is costly to run as it will create several nodes and doesn’t offer a lot of granularity on the node interactions.
As part of a Web3 Foundation Grant, we at Zondax have looked at the problem to try to come up with different solutions. We first tried to capitalize on the Polkadot conformance tests started by the foundation. Using what we learned from this first attempt we drafted a couple of recommendations: using centralized tests vectors and using the P2P interface to test nodes.
Testing through RPC endpoints
We first decided to build on the idea of the previous conformance tests tool. After talking with the maintainer, we were hinted to try to replace the FFI code by RPC calls. RPC-JSON is a standard interface which is implemented in all the blockchain projects. It would allow having a simple Python script calling those endpoints for tests. Except, it requires then the node maintainers to code and maintain those endpoints, moving the burden to the node maintainers. Writing those endpoints would demand having testing specifications agreed on too. A Proof-of-Concept for the polkadot-sdk repo was successful but we do believe there is a better approach that optimizes developers time.
Another problem we encountered was when testing the Host API. The Host API functions are being exposed to the Runtime. So for that we need the right call that would trigger a call from the Runtime a call to the Host wanted function. It is not easy to know what action triggers that call. The solution found was to write a custom Runtime that we know would call the right Host function and verify the correctness of the result. This has taken a lot of effort and can
One of our recommendations is to have a repo with centralized tests vectors that would be used by the node implementation in their tests. The most difficult part is quite often writing tests but indeed to know what to test. Based on the specification teams can collaborate on writing them. It would also be a central point where the different projects can exchange and improve how to test Polkadot protocol.
P2P testing tool
We also proposed a proof-of-concept for a tool that would this time take advantage of the P2P message interface instead of relying on RPC-JSON. This interface uses under the hood a libp2p library developed by Protocol Labs. It is the interface used by nodes to communicate between them. It is a critical part to test and it can be used to create more elaborate integration tests with a good granularity, unlike Zombienet. The pros for this solution is the flexibility, to be language agnostic and that it doesn’t need modification on the node. The PoC can be found on Zondax github: https://github.com/Zondax/polkadot-testing with a script that tests the `Ping` functionality.
Blockchain projects are complex and critical systems. They require particular attention to specification and testing, especially when there are different implementations working on the same network. We have tried to come up with a solution that would facilitate the work of developers without burdening them. Polkadot might benefit from a custom testing tool similar to what has been done for Ethereum with Hive (https://github.com/ethereum/hive), and we are looking forward to building it for the community.
Read More Articles
Feb 5, 2024
What have we learned at the Interchain Academy?
The Interchain Academy, backed by the Interchain Foundation, educates developers on the Cosmos SDK & ecosystem. With a 95% approval, it offers free, comprehensive training for building blockchain solutions.
Dec 13, 2023
Filecoin v21 Watermelon Upgrade
The Filecoin v21 “Watermelon” Network Upgrade, which happened on the 2023-12-12T13:30:00Z at epoch 3469380, introduces vital improvements and new features, aiming to enhance the network’s efficiency, security, and overall functionality.
Sep 29, 2023
Cosmos Rosetta Dockerization: Simplifying Deployment and Enhancing Accessibility
The Rosetta project, amidst blockchain's complexity, emphasizes simplicity and accessibility. Docker integration underscores this commitment, empowering users and developers alike. Rosetta, now a plug-and-play solution, paves the way for seamless blockchain innovation.
Sep 27, 2023
Sub0 2023 Lisbon: A Recap
In Lisbon, Portugal, Sub0 2023 brought together Polkadot devs and Web3 pioneers, fostering innovation and community collaboration. This annual event embodies Polkadot's essence, shaping the blockchain landscape through vibrant insights and cutting-edge ideas.
Sep 20, 2023
Looking ahead: Integrating Rosetta with Cosmos SDK v0.50
"At Zondax, we're dedicated to innovation in the ever-evolving blockchain space. Our latest move is with Cosmos SDK v0.50.0-beta.0, offering Rosetta users an enhanced, seamless experience."