Polkadot conformance testing

In this article, we will present our prototyping and research work regarding Polkadot conformance tests. Polkadot exists as a protocol and has different implementations written in different programming languages. Today, most nodes likely run the official Rust implementation. Thus, introducing a new node with a different client could lead to bugs and unwanted behaviours if it is not compliant with the specifications.Addressing this issue is a challenging problem, for which we have attempted various approaches.

Web3 Foundation has already funded projects to help testing the node:

• Web3 Foundation Polkadot conformance tests (now archived)

• Polkadot fuzzer

• Zombienet

This project relied on Foreign Function Interface (FFI) code to write unit tests for each implementation in various languages. While this approach does not require any modifications on the node part, it is heavy to maintain. It demands deep knowledge of each node's code base, and FFI code is challenging to write, read, and maintain. Development of this testing framework ceased last year.

Fuzzing is a testing technique that loads functions with randomly generated data to find unwanted behaviour. Polkadot is relying on a P2P message interface that can be accessible by anyone, including malicious actors. Testing that random inputs won’t crash a node is a good practice. However it won’t protect against human error in specification implementation or more elaborated attack.

Zombienet is a tool that spawns ephemeral Polkadot/Substrate networks on demand. It will test how an assembly of nodes would work together in a real network. It is costly to run as it will create several nodes and doesn’t offer a lot of granularity on the node interactions. 

As part of a Web3 Foundation Grant, we at Zondax have looked at the problem to try to come up with different solutions. We first tried to capitalize on the Polkadot conformance tests started by the foundation. Using what we learned from this first attempt we drafted a couple of recommendations: using centralized tests vectors and using the P2P interface to test nodes.

We first decided to build on the idea of the previous conformance tests tool. After talking with the maintainer, we were hinted to try to replace the FFI code by RPC calls. RPC-JSON is a standard interface which is implemented in all the blockchain projects. It would allow having a simple Python script calling those endpoints for tests. Except, it requires then the node maintainers to code and maintain those endpoints, moving the burden to the node maintainers. Writing those endpoints would demand having testing specifications agreed on too. A Proof-of-Concept for the polkadot-sdk repo was successful but we do believe there is a better approach that optimizes developers time.

Another problem we encountered was when testing the Host API. The Host API functions are being exposed to the Runtime. So for that we need the right call that would trigger a call from the Runtime a call to the Host wanted function. It is not easy to know what action triggers that call. The solution found was to write a custom Runtime that we know would call the right Host function and verify the correctness of the result. This has taken a lot of effort and can 

One of our recommendations is to have a repo with centralized tests vectors that would be used by the node implementation in their tests. The most difficult part is quite often writing tests but indeed to know what to test. Based on the specification teams can collaborate on writing them. It would also be a central point where the different projects can exchange and improve how to test Polkadot protocol. 

We also proposed a proof-of-concept for a tool that would this time take advantage of the P2P message interface instead of relying on RPC-JSON. This interface uses under the hood a libp2p library developed by Protocol Labs. It is the interface used by nodes to communicate between them. It is a critical part to test and it can be used to create more elaborate integration tests with a good granularity, unlike Zombienet. The pros for this solution is the flexibility, to be language agnostic and that it doesn’t need modification on the node. The PoC can be found on Zondax github: https://github.com/Zondax/polkadot-testing with a script that tests the `Ping` functionality. 

Blockchain projects are complex and critical systems. They require particular attention to specification and testing, especially when there are different implementations working on the same network. We have tried to come up with a solution that would facilitate the work of developers without burdening them. Polkadot might benefit from a custom testing tool similar to what has been done for Ethereum with Hive (https://github.com/ethereum/hive), and we are looking forward to building it for the community.