{"last_commit_sha":"c249ba93d4a52694923626b04b549cccbd869060","persisted":true,"title":"Emerald Benchmark","content":"# Emerald Benchmark ## Background The Emerald Benchmark is based on the well understood use case of international payments. The [SEPA ICT requirements](https://www.europeanpaymentscouncil.eu/document-library/rulebooks/2017-sepa-instant-credit-transfer-rulebook) provide an easily consumable set of real requirements for international payments. The key requirement is that payments must be sent, received and acknowledged back to the sender in less than 10 seconds. [Ulster Bank](https://digital.ulsterbank.ie/), a wholly owned subsidary of [Royal Bank of Scotland](https://www.rbs.com/), piloted a distributed clearing and settlement mechanism (CSM) to meet the requirements for the [major banks of Ireland](https://www.irishtimes.com/business/financial-services/banks-join-forces-to-test-new-payment-technology-1.3082772). It became known as [Emerald](https://emerald-platform.gitlab.io/). As part of this pilot they asked [GFT](https://www.gft.com/uk/en/index/) to independently test the CSM. The [results](https://emerald-platform.gitlab.io/static/emeraldTechnicalPaper.pdf) were made [public domain](https://www.finextra.com/newsarticle/29486/rbs-tests-demonstrate-ability-of-ethereum-to-support-a-national-domestic-payments-system) in order to reduce the fear, uncertainty and disinformation in late 2016 around performance and scalability of distributed ledger technologies (DLT). [Emerald code](https://gitlab.com/emerald-platform/emerald) was made open source to provide a reference implementation of a CSM using DLT. ## Aim The Emerald Benchmark aims to define a common performance test for distributed ledger technologies seeking to prove their latency, performance and scalability. This common performance test allows comparison against a number of existing international payment benchmarks. ## Principles 1. `No single point of failure` : Test must continue if any single component (including infrastructure) of the network is removed 2. `Nodes geographically distributed` : Nodes must be evenly located across different regional data centres 3. `Minimum latency is maintained` : Payments must be received and acknowledged in a minimum round trip time 4. `realistic payload` : Payment payload must use mock [ISO20022](https://www.iso20022.org/payments_messages.page) payment [pacs.008](https://www.iso20022.org/sites/default/files/documents/messages/pacs/schemas/pacs.008.001.07.xsd) and acknowledge messages [pacs.002](https://www.iso20022.org/sites/default/files/documents/messages/pacs/schemas/pacs.002.001.09.xsd) or equivalent to simulate real payments 5. `bi-directional payments` : All Payment nodes must send and receive payments randomly to other nodes in the network ## Measures | Measure | Description | Units | | :------: | :------: | :------: | | `Latency` | seconds to receive a transaction's acknowledgement (ack) | seconds | | `Rate` | transaction acks received per second | tx/sec | | `Scale` | number of nodes creating transactions | nodes | | `Throughput` | Rate x Scale | tx/sec | ## Results In order to ease comparison we would recommend the following visuals: * Graph showing Average Rate vs Scale (where minimum latency is maintained) * Graph showing Scale, Latency and Rate (average, min and max) vs time ## Test script The test is based on a set of geographically distributed nodes sending payments to one another. Each payment must be sent and an acknowledgement received back to the sender in a minimum round trip time (latency). The test should only increase the rate at which payments are sent once the minimum latency can be maintained. Parameters - MinLatency, MinRate, MaxRate, MaxScale Increase number of nodes to MaxScale For each node: ``` parameters - MinLatency, MinRate, MaxRate Rate = MinRate Send transaction messages every Rate seconds to another random node on the network On receiving transaction ack: latency = transaction ack received timestamp - transaction sent timestamp record latency and rate if latency > MinLatency then decrease Rate but not lower than MinRate else increase Rate but not higher than MaxRate ``` ## Test scenarios | Name | MinLatency | MinRate | MaxRate | MaxScale | Throughput | Badge | | :------: | :------: | :------: | :------: | :------: | :------: | :------: | | minimum | <10 | 1 | 1 | 3 | 3 | | | `deca` | <10 | 1 | 1 | 10 | 10 | ![](https://img.shields.io/badge/emerald-deca-green.svg) | | `hecto` | <10 | 1 | 10 | 10 | 100 | ![Emerald badge](https://img.shields.io/badge/emerald-hecto-green.svg) | | `kilo` | <10 | 10 | 100 | 10 | 1,000 | ![Emerald badge](https://img.shields.io/badge/emerald-kilo-green.svg) | | `mega` | <10 | 100 | 1,000 | 1,000 | 1 million | ![Emerald badge](https://img.shields.io/badge/emerald-mega-green.svg) | | `giga` | <10 | 1,000 | 100,000 | 10,000 | 1 billion | ![Emerald badge](https://img.shields.io/badge/emerald-giga-green.svg) | A project should show the badge if it can evidence completing one of the above scenarios. eg a green ![Emerald badge](https://img.shields.io/badge/emerald-mega-green.svg) should be shown if the test showed evidence of a `Scale` and 'Rate' of >1000 while maintaining a minimum `Latency` of < 10 seconds. Conversely the red badge ![Emerald badge](https://img.shields.io/badge/emerald-mega-red.svg) should be shown if the test showed evidence of a `Scale` and `Rate` of <1000 or the minimum `Latency` failed to maintain <10 seconds. ## Benchmarks | Name | Latency | Rate | Scale | Throughput | Badge | | :------: | :------: | :------: | :------: | :------: | :------: | | Emerald (Irish pilot) | <10 | 100 | 6 | 600 | ![Emerald badge](https://img.shields.io/badge/emerald-hecto-green.svg) | | BitCoin | `>10` | 7 | 12,000 | 84k | ![Emerald badge](https://img.shields.io/badge/emerald-kilo-red.svg) | | SEPA ICT (EBA STEP2) | <10 | 250 | 1200 | 300k | ![Emerald badge](https://img.shields.io/badge/emerald-kilo-green.svg) | | Ethereum (main net) | `>10` | 16 | 25,000 | 400k | ![Emerald badge](https://img.shields.io/badge/emerald-kilo-red.svg) | | Ripple (XRP) | <10 | 1500 | 500 | 750k | ![Emerald badge](https://img.shields.io/badge/emerald-kilo-green.svg) | | SWIFT (FIN) | <10 | 400 | 11,000 | 4.4m | ![Emerald badge](https://img.shields.io/badge/emerald-mega-green.svg) | | Visa | <10 | 65,000 | 16,300 | 1.1bn | ![Emerald badge](https://img.shields.io/badge/emerald-giga-green.svg) | Data correct @ Jan 2018 Sources: * https://www.swift.com/about-us/swift-fin-traffic-figures * https://www.ebaclearing.eu/services/step2-sct/statistics/ * https://usa.visa.com/dam/VCOM/download/corporate/media/visanet-technology/aboutvisafactsheet.pdf * https://ripple.com/xrp/ * https://bitinfocharts.com/comparison/ethereum-transactions.html * https://blockchain.info/charts/n-transactions ## Reference Architecture The following reference architecture shows how an Emerald Test can be performed on a single cloud provider with regional data centres. * Test Executor - Allows control of the start/stop/scaling of nodes * Cloud Portal - Manages the cloud infrastructure * Test Reporter - Provides reports on results * Emerald Test - Implementation of test script described above, records to logging/monitoring component ![EmeraldTest](/uploads/afdf468ae182fb5214cfe5d5a77383db/EmeraldTest.png) ","front_matter":{},"format":"markdown","uploads_path":"https://gitlab.com/api/v4/projects/1290080/wikis/attachments","slug":"Emerald-Benchmark","path":"/emerald-platform/emerald/-/wikis/Emerald-Benchmark","wiki_path":"/emerald-platform/emerald/-/wikis/home","help_path":"/help/user/project/wiki/_index.md","markdown_help_path":"/help/user/markdown.md","markdown_preview_path":"/emerald-platform/emerald/-/wikis/Emerald-Benchmark/preview_markdown","create_path":"/emerald-platform/emerald/-/wikis","container_full_path":"emerald-platform/emerald"}