Blog

/

Polkadot conformance testing

Polkadot conformance testing

December 22, 2023

3 min read

Protocol


We discuss Polkadot conformance tests and their importance. Polkadot, primarily using a Rust implementation, faces challenges with new node introductions. We explore various testing approaches, including W3F Polkadot conformance tests, Polkadot fuzzer, and Zombienet.

In this article, we will present our prototyping and research work regarding Polkadot conformance tests. Polkadot exists as a protocol and has different implementations written in different programming languages. Today, most nodes likely run the official Rust implementation. Thus, introducing a new node with a different client could lead to bugs and unwanted behaviours if it is not compliant with the specifications.Addressing this issue is a challenging problem, for which we have attempted various approaches.

Polkadot conformance tests existing work


Web3 Foundation has already funded projects to help testing the node:

  • Web3 Foundation Polkadot conformance tests (now archived)

  • Polkadot fuzzer

  • Zombienet

Web3 Foundation Polkadot conformance tests


This project relied on Foreign Function Interface (FFI) code to write unit tests for each implementation in various languages. While this approach does not require any modifications on the node part, it is heavy to maintain. It demands deep knowledge of each node's code base, and FFI code is challenging to write, read, and maintain. Development of this testing framework ceased last year.

Polkadot fuzzer


Fuzzing is a testing technique that loads functions with randomly generated data to find unwanted behaviour. Polkadot is relying on a P2P message interface that can be accessible by anyone, including malicious actors. Testing that random inputs won’t crash a node is a good practice. However it won’t protect against human error in specification implementation or more elaborated attack.

Zombienet


Zombienet is a tool that spawns ephemeral Polkadot/Substrate networks on demand. It will test how an assembly of nodes would work together in a real network. It is costly to run as it will create several nodes and doesn’t offer a lot of granularity on the node interactions. 

Zondax’s work


As part of a Web3 Foundation Grant, we at Zondax have looked at the problem to try to come up with different solutions. We first tried to capitalize on the Polkadot conformance tests started by the foundation. Using what we learned from this first attempt we drafted a couple of recommendations: using centralized tests vectors and using the P2P interface to test nodes.

Testing through RPC endpoints


We first decided to build on the idea of the previous conformance tests tool. After talking with the maintainer, we were hinted to try to replace the FFI code by RPC calls. RPC-JSON is a standard interface which is implemented in all the blockchain projects. It would allow having a simple Python script calling those endpoints for tests. Except, it requires then the node maintainers to code and maintain those endpoints, moving the burden to the node maintainers. Writing those endpoints would demand having testing specifications agreed on too. A Proof-of-Concept for the polkadot-sdk repo was successful but we do believe there is a better approach that optimizes developers time.

Another problem we encountered was when testing the Host API. The Host API functions are being exposed to the Runtime. So for that we need the right call that would trigger a call from the Runtime a call to the Host wanted function. It is not easy to know what action triggers that call. The solution found was to write a custom Runtime that we know would call the right Host function and verify the correctness of the result. This has taken a lot of effort and can 

Tests vectors


One of our recommendations is to have a repo with centralized tests vectors that would be used by the node implementation in their tests. The most difficult part is quite often writing tests but indeed to know what to test. Based on the specification teams can collaborate on writing them. It would also be a central point where the different projects can exchange and improve how to test Polkadot protocol. 

P2P testing tool


We also proposed a proof-of-concept for a tool that would this time take advantage of the P2P message interface instead of relying on RPC-JSON. This interface uses under the hood a libp2p library developed by Protocol Labs. It is the interface used by nodes to communicate between them. It is a critical part to test and it can be used to create more elaborate integration tests with a good granularity, unlike Zombienet. The pros for this solution is the flexibility, to be language agnostic and that it doesn’t need modification on the node. The PoC can be found on Zondax github: https://github.com/Zondax/polkadot-testing with a script that tests the `Ping` functionality. 

Conclusion


Blockchain projects are complex and critical systems. They require particular attention to specification and testing, especially when there are different implementations working on the same network. We have tried to come up with a solution that would facilitate the work of developers without burdening them. Polkadot might benefit from a custom testing tool similar to what has been done for Ethereum with Hive (https://github.com/ethereum/hive), and we are looking forward to building it for the community.

Read More


The Evolution of Transaction Signing in Cosmos: Introducing Sign Mode Textual

Sign Mode Textual introduces a secure, user-friendly method for transaction signing with hardware devices, advancing efficiency and security in the Cosmos SDK.


Making the Most Out of the Beryx API: Your Ally for Filecoin Development

For over a year, Zondax has developed the Beryx Explorer and API. With over 90 endpoints, it streamlines interactions with the FVM and FEVM.


Experiencing LaBitConf: An evening of Crypto, and the Argentinian Community

LaBitConf set the stage for our side event with Filecoin Orbit at Crypster Club. Nearly 50 attendees enjoyed Web3 talks, games, and Argentine treats like fernet and medialunas. Insights from Filecoin and Zondax made it memorable.

Other Articles


672934960cbb3e00aac14e38

November 4, 2024

Unlocking Your Zcash: A Step-by-Step Guide to Using the Zcash Shielded App on Ledger

Unlock and manage your Zcash securely with the Zondax-developed Zcash Shielded App for Ledger. This step-by-step guide covers everything from setting up your environment to handling transactions, ensuring privacy and control. Dive into shielded Zcash today!

66a7b04d183d7f00aa1afb83

July 29, 2024

Announcing Lotus v1.28.1: Elevating Filecoin to Network Version 23 (Waffle) 🧇

Filecoin's Lotus v1.28.1, codenamed Waffle, introduces Network Version 23, enhancing features and performance. Upgrade on 2024-08-06 at 12:00 UTC. Key updates include FIP improvements and Fast Finality.

66e22a1519115200a95ca27c

Integration

September 24, 2024

Spacemesh Is Live

The Spacemesh Ledger App is live! Manage your crypto with top-notch security, designed by Zondax, to keep your assets safe in the growing Spacemesh ecosystem.

Announcing Lotus v1.28.1: Elevating Filecoin to Network Version 23 (Waffle) 🧇

Filecoin's Lotus v1.28.1, codenamed Waffle, introduces Network Version 23, enhancing features and performance. Upgrade on 2024-08-06 at 12:00 UTC. Key updates include FIP improvements and Fast Finality.

66a7b04d183d7f00aa1afb83

Integration

Spacemesh Is Live

The Spacemesh Ledger App is live! Manage your crypto with top-notch security, designed by Zondax, to keep your assets safe in the growing Spacemesh ecosystem.

66e22a1519115200a95ca27c