test: integration tests for RPC gateway (#137)

* feat: add config parameters for fine-grained routing control (#126)

* Added instructions to `README.md`

* feat: update dependencies

* feat: updated dependencies

* feat: new mockery version

* feat: add routing config params

* feat: upgrade Go version

* fix: suppress legacy linter errors

* fix: field alignments in structs

* fix: field linter errors

* fix: linter errors

* feat: add routing config params

* feat: validate routing error rate

* feat: refactoring

* feat: add a TODO

* feat: `ErrorsConfig` and `LatencyConfig` are now pointers

* feat: add a field to routing config test

* feat: set defaults for `DetectionWindow` and `BanWindow`

* feat: move RoutingConfig functions closer to the struct

* feat: changed `or` to `and`

* feat: split function into validation and setting defaults components

* feat: settings the config defaults now mirrors the validation workflow

* feat: remove redundant call

* feat: upgrade Go version to 1.22

* feat: config parsing enhancements

* feat: latency checker

* feat: latency checker wiring

* feat: support `alwaysRoute` option

* fix: the default threshold is now set in LatencyConfig if not specified

* fix: getLatencyThresholdForMethod

* fix: rename constant

* fix: rename struct field

* fix: ShouldRunPassiveHealthChecks field is now set correctly

* feat: add method as a Prometheus label

* fix: remove extra RecordRequest call

* fix: rename method

* fix: check if we should run passive check before initializing the client

* fix: remove ShouldRun check in IsLatencyAcceptable.Apply method

* feat: add logging

* fix: add TODOs

* fix: updated comments

* fix: add TODOs

* fix: rename LatencyCheck to ErrorLatencyCheck

* fix: rename `latency` module to `errorlatency`

* fix: passive health check is no longer initialized if it is disabled

* fix: improve a metric's help text

* fix: rename metric variable names

* fix: typo

* fix: remove confusing "TODOs"

* fix: better phrasing

* fix: lint

* fix: add panics

* fix: receiver cannot be nil

* feat: check if error/latency routing control is enabled

* feat: expand a test

* fix: chain routing config is no longer initialized if global routing config is not specified

* feat: add tests

* fix: improve readability

* fix: improve readability

* fix: remove a TODO

* fix: improve code readability

* fix: improve code readability

* fix: add a comment for `GlobalConfig::setDefaults` method

* fix: rename IsEnhancedRoutingControlEnabled method

* fix: rename HasEnhancedRoutingControlDefined method

* feat: disable BlockHeightCheck

* fix: bug fix

* fix: latency threshold determination for method

* fix: lint

* fix: return top-level latency threshold

* fix: move code block

* fix: adjust MinNumRequestsForRate

* feat: test servers

* fix: add a TODO

* fix: expand a TODO

* fix: skip failing tests

* fix: add an ignored directory

* feat: add a test config file

* fix: servers are now terminated correctly

* feat: start instructions file on testing

* feat: start a file containing common shell functions

* feat: start the first test script

* fix: rename a constant

* fix: remove an unused function

* fix: error rate recording

* feat: test for error rate routing

* feat: add command to run error based routing tests

* feat: add tests for error string based routing

* feat: add tests for error string and HTTP code based routing

* feat: add tests for error string and HTTP code based routing

* feat: add tests for latency based routing

* feat: add support for testing with the `alwaysRoute` option

* fix: test & fix ban window

* fix: remove a TODO

* fix: field alignment

* fix: field alignment

* fix: readability

* fix: update `README.md`

* feat: initial implementation of `AlwaysRouteRoutingStrategy`

* fix: comment formatting

* fix: comment phrasing

* fix: add a TODO

* fix: renaming

* fix: revert to the previous version of `PriorityRoundRobinStrategy`

* fix: remove `HealthStatus` field

* feat: LatencyChecker can now be configured to only check latency, errors, or both

* feat: create separate instances of LatencyChecker for error and latency checking

* feat: plumbing for `AlwaysRouteFilteringStrategy`

* fix: tests

* fix: removable filters

* feat: add debug logging

* fix: move `getFilterTypeName` function to `node_filter` module

* refactor: deduplicate filter upstreams code

* refactor: remove passive error/latency checker code

* refactor: split error latency checker into two structs

* fix: error and latency filters are now passed to regular filtering routing strategy

* fix: typo

* style: remove unused global variable

* test: add test

* refactor: test

* refactor: test

* refactor: add test

* test: add test

* test: add high latency test

* refactor: routing config struct

* test: add latency test with always route

* test: add error rate test with always route

* fix: change hardcoded constant

* test: add expected output

* test: update expected output

* test: update expected output

* docs: update instructions

* test: add expected output

* refactor: move integration test files under subdir

* test: fix readme and generate expected output

* fix: revert changes to production files

.gitignore

@@ -22,4 +22,4 @@ build/
 .idea
 
 # Temporary files
-/__pycache__/
+__pycache__/

internal/checks/common.go

@@ -10,7 +10,7 @@ import (
 
 const (
 	PercentPerFrac        = 100
-	MinNumRequestsForRate = 4 // The minimum number of requests required to compute the error rate.
+	MinNumRequestsForRate = 3 // The minimum number of requests required to compute the error rate.
 )
 
 // NewCircuitBreaker abstracts away the rather complex circuitbreaker.Builder API.

tests-integration/TESTING.md

@@ -0,0 +1,53 @@
+## Setup
+
+All commands must be run from this directory, except RPC gateway, which must be run from the repository root.
+
+After running a test, `git diff` shows the difference between the expected output and the actual output.
+If the diff is empty, the test passed.
+
+```zsh
+cd tests-integration
+
+# Start test servers.
+# WARNING: Ports are not immediately available after stopping servers via Ctrl+C.
+#  Wait a minute or so before starting them again.
+./start_servers.sh
+```
+
+**You must restart the gateway after each test run to clear the state!**
+
+## Testing With Routing Control Disabled
+
+```zsh
+LOG_LEVEL=debug go run cmd/gateway/main.go tests-integration/configs/config-disabled.yml
+
+./scripts/http-error.sh > expected-output/routing-disabled/http-error.txt
+./scripts/error-string.sh > expected-output/routing-disabled/error-string.txt
+./scripts/error-string-and-http.sh > expected-output/routing-disabled/error-string-and-http.txt
+./scripts/latency.sh > expected-output/routing-disabled/latency.txt
+./scripts/latency-override.sh > expected-output/routing-disabled/latency-override.txt
+````
+
+## Testing With Routing Control Enabled
+
+```zsh
+LOG_LEVEL=debug go run cmd/gateway/main.go tests-integration/configs/config.yml
+
+./scripts/http-error.sh > expected-output/routing-enabled/http-error.txt
+./scripts/error-string.sh > expected-output/routing-enabled/error-string.txt
+./scripts/error-string-and-http.sh > expected-output/routing-enabled/error-string-and-http.txt
+./scripts/latency.sh > expected-output/routing-enabled/latency.txt
+./scripts/latency-override.sh > expected-output/routing-enabled/latency-override.txt
+```
+
+## Testing With Routing Control Enabled and `alwaysRoute` Option
+
+```zsh
+LOG_LEVEL=debug go run cmd/gateway/main.go tests-integration/configs/config-always-route.yml
+
+./scripts/http-error.sh > expected-output/routing-enabled-always-route/http-error.txt
+./scripts/error-string.sh > expected-output/routing-enabled-always-route/error-string.txt
+./scripts/error-string-and-http.sh > expected-output/routing-enabled-always-route/error-string-and-http.txt
+./scripts/latency.sh > expected-output/routing-enabled-always-route/latency.txt
+./scripts/latency-override.sh > expected-output/routing-enabled-always-route/latency-override.txt
+```

tests-integration/configs/config-always-route.yml

@@ -0,0 +1,46 @@
+global:
+  port: 8080
+  routing:
+    banWindow: 3s
+    errors:
+      rate: 0.25
+      httpCodes:
+        - 404
+      errorStrings:
+        - "rate limit exceeded"
+    latency:
+      threshold: 100ms
+      methods:
+        - method: eth_slowMethod
+          threshold: 2000ms
+    alwaysRoute: true
+
+chains:
+  - chainName: mainnet
+    groups:
+      - id: primary
+        priority: 0
+      - id: fallback
+        priority: 1
+
+    upstreams:
+      - id: zero1
+        httpURL: http://localhost:3333/any/path
+        group: primary
+        nodeType: archive
+        healthCheck:
+          skipPeerCountCheck: true
+
+      - id: zero2
+        httpURL: http://localhost:4444/any/path
+        group: primary
+        nodeType: archive
+        healthCheck:
+          skipPeerCountCheck: true
+
+      - id: one
+        httpURL: http://localhost:5555/any/path
+        group: fallback
+        nodeType: archive
+        healthCheck:
+          skipPeerCountCheck: true

tests-integration/configs/config-disabled.yml

@@ -0,0 +1,32 @@
+global:
+  port: 8080
+
+chains:
+  - chainName: mainnet
+    groups:
+      - id: primary
+        priority: 0
+      - id: fallback
+        priority: 1
+
+    upstreams:
+      - id: zero1
+        httpURL: http://localhost:3333/any/path
+        group: primary
+        nodeType: archive
+        healthCheck:
+          skipPeerCountCheck: true
+
+      - id: zero2
+        httpURL: http://localhost:4444/any/path
+        group: primary
+        nodeType: archive
+        healthCheck:
+          skipPeerCountCheck: true
+
+      - id: one
+        httpURL: http://localhost:5555/any/path
+        group: fallback
+        nodeType: archive
+        healthCheck:
+          skipPeerCountCheck: true

tests-integration/configs/config.yml

@@ -0,0 +1,45 @@
+global:
+  port: 8080
+  routing:
+    banWindow: 3s
+    errors:
+      rate: 0.25
+      httpCodes:
+        - 404
+      errorStrings:
+        - "rate limit exceeded"
+    latency:
+      threshold: 100ms
+      methods:
+        - method: eth_slowMethod
+          threshold: 2000ms
+
+chains:
+  - chainName: mainnet
+    groups:
+      - id: primary
+        priority: 0
+      - id: fallback
+        priority: 1
+
+    upstreams:
+      - id: zero1
+        httpURL: http://localhost:3333/any/path
+        group: primary
+        nodeType: archive
+        healthCheck:
+          skipPeerCountCheck: true
+
+      - id: zero2
+        httpURL: http://localhost:4444/any/path
+        group: primary
+        nodeType: archive
+        healthCheck:
+          skipPeerCountCheck: true
+
+      - id: one
+        httpURL: http://localhost:5555/any/path
+        group: fallback
+        nodeType: archive
+        healthCheck:
+          skipPeerCountCheck: true

tests-integration/expected-output/routing-disabled/error-string-and-http.txt

@@ -0,0 +1,22 @@
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_string_404, secs=0, error=404\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_string_404, secs=0, error=404\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_string_404, secs=0, error=404\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+<html><body><h1>POST request received: port=4444, method=eth_anotherMethod, secs=0, error=</h1></body></html>
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_string_another_error, secs=0, error=another error\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_400, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 400"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+<html><body><h1>POST request received: port=4444, method=eth_method, secs=0, error=</h1></body></html>
+

tests-integration/expected-output/routing-disabled/error-string.txt

@@ -0,0 +1,22 @@
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+<html><body><h1>POST request received: port=4444, method=eth_anotherMethod, secs=0, error=</h1></body></html>
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_string_another_error, secs=0, error=another error\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+<html><body><h1>POST request received: port=4444, method=eth_method, secs=0, error=</h1></body></html>
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_string_rate_limit_exceeded, secs=0, error=rate limit exceeded\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 403"}
+
+<html><body><h1>POST request received: port=4444, method=eth_method, secs=0, error=</h1></body></html>
+

tests-integration/expected-output/routing-disabled/http-error.txt

@@ -0,0 +1,46 @@
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+<html><body><h1>POST request received: port=4444, method=eth_anotherMethod, secs=0, error=</h1></body></html>
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_400, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 400"}
+
+<html><body><h1>POST request received: port=4444, method=eth_method, secs=0, error=</h1></body></html>
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+<html><body><h1>POST request received: port=4444, method=eth_method, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_method, secs=0, error=</h1></body></html>
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=4444, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+{"message":"Request could not be routed, err: error making request to origin. err: \u003cnil\u003e, resp: \u003chtml\u003e\u003cbody\u003e\u003ch1\u003ePOST request received: port=3333, method=eth_404, secs=0, error=\u003c/h1\u003e\u003c/body\u003e\u003c/html\u003e, respCode: 404"}
+
+<html><body><h1>POST request received: port=4444, method=eth_method, secs=0, error=</h1></body></html>
+

tests-integration/expected-output/routing-disabled/latency-override.txt

@@ -0,0 +1,24 @@
+<html><body><h1>POST request received: port=4444, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_slowMethod, secs=0, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_slowMethod, secs=0, error=</h1></body></html>
+

tests-integration/expected-output/routing-disabled/latency.txt

@@ -0,0 +1,32 @@
+<html><body><h1>POST request received: port=4444, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_77, secs=0.077, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_77, secs=0.077, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_177, secs=0.177, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_77, secs=0.077, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_77, secs=0.077, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=4444, method=eth_77, secs=0.077, error=</h1></body></html>
+
+<html><body><h1>POST request received: port=3333, method=eth_77, secs=0.077, error=</h1></body></html>
+