Proxy Verifier

Proxy Verifier is an HTTP replay tool designed to verify the behavior of HTTP proxies. It builds a verifier-client binary and a verifier-server binary which each read a set of YAML files (or JSON files, since JSON is YAML) that specify the HTTP traffic for the two to exchange.

Proxy Verifier supports the HTTP replay of the following protocols:

HTTP/1.x and HTTP/2
HTTP and HTTP over TLS
IPv4 and IPv6

Broadly speaking, Proxy Verifier is designed to address two proxy testing needs:

Traffic correctness testing: In this context Proxy Verifier can be used in manual or automated end to end tests to verify correct behavior of the details of generally a small number of transactions. Transaction Box is an example of a tool that relies entirely on Proxy Verifier for its automated end to end tests (see its autest directory).
Production simulation testing: In this context, Proxy Verifier is used in an isolated lab environment configured as much like a production environment as possible. The Verifier client and server are provided replay files as input that are either auto generated or collected via a tool like Traffic Dump from an actual production environment. Proxy verifier then replays this production-like traffic against the proxy under test. Proxy Verifier can replay such traffic at rates over 10k requests per second. Here are some examples where this use of Proxy Verifier can be helpful:
- Safely testing experimental versions of a patch.
- Running diagnostic versions of the proxy that may not be performant enough for the production environment. Debug, Valgrind, and ASan builds are examples of build configurations whose resultant proxy performance may be impossible to run in production but can be run safely in the production simulation via Proxy Verifier.
- Performance comparison across versions of the proxy. Since Proxy Verifier replays the traffic for the same set of replay files consistently across runs, different versions of the proxy can be compared with regard to their performance against the same replay dataset. This affords the ability to compare detect performance improvement or degradation across versions of the proxy.

It should be noted that when using Proxy Verifier in a production simulation environment, the proxy will be receiving traffic from the client that looks like production traffic. Thus, by design, the HTTP request targets and Host header fields will reference actual production servers. It will be important to configure the proxy under test to direct these requests not to the actual production servers but to the Proxy Verifier server or servers. The way this is achieved will depend upon the proxy. One way to accomplish this is to configure the production environment with a test DNS server such as MicroDNS configured to resolve all host names to the Proxy Verifier server or servers in the production simulation environment.

Traffic Replay Specification

HTTP Specification

Proxy Verifier traffic behavior is specified via YAML files. The behavior for each connection is specified under the top-most sessions node, the value of which is a sequence where each item in the sequence describes the characteristics of each connection. For each session the protocol stack can be specified via the protocol node. This node describes the protocol characteristics of the connection such as what version of HTTP to use, whether to use TLS and what the characteristics of the TLS handshake should be. In the absence of a protocol node the default protocol is HTTP/1 over TCP. See Protocol Specification below for details about the protocol node. Each session is run in parallel by the verifier-client (although see --thread-limit <number> below for a way to serialize them).

In addition to protocol specification, each session in the sessions sequence contains a transactions node. This itself is a sequence of transactions that should be replayed for the associated session. Within each transaction, Proxy Verifier's traffic replay behavior is specified in the client-request and server-response nodes. client-request nodes are used by the Proxy Verifier client and tell it what kind of HTTP requests it should generate. server-response nodes are used by the Proxy Verifier server to indicate what HTTP responses it should generate. Proxy traffic verification behavior is described in the proxy-request and proxy-response nodes which will be covered in Traffic Verification Specification. For HTTP/1, these transactions are run in sequence for each session; for HTTP/2, the transactions are run in parallel.

For HTTP/1 requests, client-request has a map as a value which contains each of the following key/value YAML nodes:

method: This takes the HTTP method as the value, such as GET or POST.
url: This takes the request target as the value, such as /a/path.asp or https://www.example.com/a/path.asp.
version: This takes the HTTP version, such as 1.1 or 2.
headers: This takes a fields node which has, as a value, a sequence of HTTP fields. Each field is itself a sequence of two values: the name of the field and the value for the field.
content: This specifies the body to send. It takes a map as a value. The user can specify a size integer value in which an automated body of that size will be generated. Otherwise a data string value can be provided in which the specified body will be sent and an encoding taking plain or uri to specify that the string is raw or URI encoded.

Here's an example of a client-request node describing an HTTP/1.1 POST request with a request target of /pictures/flower.jpeg and containing four header fields with a body size of 399 bytes:

  client-request:
    method: POST
    url: /pictures/flower.jpeg
    version: '1.1'
    headers:
      fields:
      - [ Host, www.example.com ]
      - [ Content-Type, image/jpeg ]
      - [ Content-Length, '399' ]
      - [ uuid, 1234 ]
    content:
        size: 399

For convenience, if Proxy Verifier detects a Content-Length header, then the content node stating the size of the body is not required. Thus this can be simplified to:

  client-request:
    method: POST
    url: /pictures/flower.jpeg
    version: '1.1'
    headers:
      fields:
      - [ Host, www.example.com ]
      - [ Content-Type, image/jpeg ]
      - [ Content-Length, '399' ]
      - [ uuid, 1234 ]

server-response nodes indicate how the Proxy Verifier server should respond to HTTP requests. In place of method, url, and version, HTTP/1 responses take the following nodes:

status: This takes an integer corresponding to the HTTP response status, such as 200 or 404.
reason: This takes a string that describes the status, such as "OK" or "Not Found".

Here's an example of an HTTP/1 server-response with a status of 200, four fields, and a body of size 3,432 bytes:

  server-response:
    status: 200
    reason: OK
    headers:
      fields:
      - [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
      - [ Content-Type, image/jpeg ]
      - [ Transfer-Encoding, chunked ]
      - [ Connection, keep-alive ]
    content:
      size: 3432

Observe in this case that the response contains the Transfer-Encoding: chunked header field, indicating that the body should be chunk encoded. Proxy Verifier supports chunk encoding for both requests and responses and will use this when it detects the Transfer-Encoding: chunked header field. In this case, the 3,432 bytes will be the size of the chunked body payload without the bytes for the chunk protocol (chunk headers, etc.).

Finally, here is an example of a response with specific body content sent (YAML in this case) as opposed to the generated content specified by the content:size nodes above:

  server-response:
    status: 200
    reason: OK
    headers:
      fields:
      - [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
      - [ Content-Type, text/yaml ]
      - [ Transfer-Encoding, chunked ]
      - [ Connection, keep-alive ]
    content:
      encoding: plain
      data: |
          ### Heading

          * Bullet
          * Points

Server Response Lookup

The client-request and server-response nodes are all that is required to direct Proxy Verifier's replay of a transaction. The next section will describe how to specify proxy transaction verification behavior via the proxy-request and proxy-response nodes. Before proceeding to that, however, it is valuable to understand how the Verifier server decides which response to send for any given request it receives. Consider again the client-request node in the preceding example:

  client-request:
    method: POST
    url: /pictures/flower.jpeg
    version: '1.1'
    headers:
      fields:
      - [ Host, www.example.com ]
      - [ Content-Type, image/jpeg ]
      - [ Content-Length, '399' ]
      - [ uuid, 1234 ]
    content:
      size: 399

Note the presence of the uuid field. A field such as this is sent by the client and used by the server. When a server receives a request, keep in mind that all it has to direct its behavior is what it has parsed from the replay file or files and what it sees in the request. There may be hundreds of thousands of parsed server-response nodes, each describing a unique response with which the server may reply for any given incoming request. The server does not talk directly to the client, so it cannot communicate with it and say, "Hey Verifier client, I just read such and such request off the wire. Which response would you like me to send for this?" When it receives an HTTP request, therefore, it only has the contents of that request from which to choose its response. To facilitate the server's behavior, a unique key is associated with every request. During the YAML parsing phase, the Verifier server associates each parsed server-response with a key it derives from the associated client-request for the transaction. During the traffic replay phase, when it reads a request off the wire, it derives a key from the HTTP request using the same process for generating keys from client-request nodes used in the parsing phase. It then looks up the proper YAML-specified response using that key. By default, Proxy Verifier uses the uuid HTTP header field values as the key, as utilized in the examples in this document, but this is configurable via the --format command line argument. For details see the section describing this argument below.

Both the client and the server will fail the YAML parsing phase and exit with a non-zero return if either parses a transaction for which they cannot derive a key. If during the traffic processing phase the Verifier server somehow receives a request for which it cannot derive a key, it will return a 404 Not Found response and close the connection upon which it received the request.

HTTP/2 Specification

For HTTP/2, the protocol describes the initial request line values with pseudo header fields. For that protocol, therefore, the user need not specify the method, url, and version nodes and would instead specify these values more naturally as pseudo headers in the fields sequence. Here is an example of an HTTP/2 client-request analogous to the HTTP/1 request above (note that the Content-Length header field is optional in HTTP/2 and as such is omitted here):

  client-request:
    headers:
      fields:
      - [ :method, POST ]
      - [ :scheme, https ]
      - [ :authority, www.example.com ]
      - [ :path, /pictures/flower.jpeg ]
      - [ Content-Type, image/jpeg ]
      - [ uuid, 1234 ]
    content:
      size: 399

The status code is described in the :status pseudo header field. Also, HTTP/2 does not allow for chunk encoding nor does it use the Connection header field, using instead its framing mechanism to describe the body and session lifetime. An analogous HTTP/2 response to the HTTP/1 request above, therefore, would look like the following:

  server-response:
    headers:
      fields:
      - [ :status, 200 ]
      - [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
      - [ Content-Type, image/jpeg ]
    content:
      size: 3432

Trailer response headers are useful for transmitting additional fields after the message body, providing dynamically generated metadata such as integrity checks or post-processing status. Proxy Verifier supports sending and receiving trailer headers, as demonstrated below:

  server-response:
    status: 200
    headers:
      fields:
      # Some fields...
    content:
      # Some data...
    trailers:
      encoding: esc_json
      fields:
      - [ x-test-trailer-1, one ]
      - [ x-test-trailer-2, two ]

It is also possible to specify everything as a sequence of frames. The available options for the frame sequence are:

DATA
HEADERS
RST_STREAM
GOAWAY

Here's an example replay file:

  client-request:
    frames:
    - HEADERS:
        headers:
          fields:
          - [ :method, POST ]
          - [ :scheme, https ]
          - [ :authority, www.example.com ]
          - [ :path, /pictures/flower.jpeg ]
          - [ Content-Type, image/jpeg ]
          - [ uuid, 1234 ]
    - DATA:
        content:
          size: 399
    - RST_STREAM:
          error-code: STREAM_CLOSED

  server-response:
    frames:
    - HEADERS:
        headers:
          fields:
          - [ :status, 200 ]
          - [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
          - [ Content-Type, image/jpeg ]
    - DATA:
        content:
          size: 3432

HEADERS and DATA frame

The HEADERS and DATA frame nodes are designed to be specified in a way that is consistent with their HTTP/1 counterparts. From a parsing perspective, this means they simply wrap the headers and content nodes that are used for HTTP/1 specification as described above. For an example, see the RST_STREAM frame section below.

Note that multiple DATA frames can be specified for requests and responses. The replay follows the same order the DATA frames are listed. See the example below:

  client-request:
    frames:
    - HEADERS:
        headers:
          fields:
          - [:method, POST]
          - [:scheme, https]
          - [:authority, example.data.com]
          - [:path, /a/path]
          - [Content-Type, text/html]
          - [uuid, 1]
    - DATA:
          content:
          encoding: plain
          data: client_data_1
    - DATA:
        content:
          encoding: plain
          data: client_data_2

RST_STREAM frame

In some cases, there might be a need to test the behavior of the proxy when the client or server terminates the stream, either because of an unexpected error or intentionally cancelling the stream. In HTTP/2, peers terminate a transaction by sending a RST_STREAM frame which includes an error code. In the replay file, this is specified via a RST_STREAM frame node which is ordered within the stream to indicate when it should be sent and includes an error-code node to specify which error code should be used.

The following sample replay file snippet demonstrate how to specify such a test scenario:

sessions:
- protocol:
  - name: http
    version: 2
  - name: tls
    sni: test_sni
  - name: tcp
  - name: ip
    version: 4
  transactions:
  - client-request:
      frames:
      - HEADERS:
          headers:
            fields:
            - [:method, POST]
            - [:scheme, https]
            - [:authority, example.data.com]
            - [:path, /a/path]
            - [Content-Type, text/html]
            - [Content-Length, '11']
            - [uuid, 1]
      - RST_STREAM:
          error-code: INTERNAL_ERROR
      - DATA:
          content:
            encoding: plain
            data: client_test
            size: 11

Note that this example specifies the following:

A sequence of frames to be sent under the client-request node. The order of the frames listed is the order of the frames that Proxy Verifier will send during the traffic replay of the stream.
Thus, in this case, the client terminates the stream after sending the HEADERS frame since the RST_STREAM is specified after the HEADERS frame in the sequence.
The client terminates with the error code INTERNAL_ERROR, specified by error-code under the RST_STREAM frame.

Be aware that when Proxy Verifier generates HTTP/2 frames, it determines from the frames elements where to insert the END_STREAM flag. If a request or response just has a HEADERS frame, then the END_STREAM flag will be added to the end of the HEADERS frame. If there are both HEADERS and DATA frames, then the END_STREAM will be placed after the DATA frame. Proxy Verifier does not, however, use RST_STREAM frames to influence the END_STREAM flag. Thus in the above example, having the RST_STREAM between the HEADERS and DATA frames means that the HEADERS will not have an END_STREAM flag before the RST_STREAM is sent. For this reason, you must include a DATA frame after a RST_STREAM frame, even though the DATA frame will not be sent, in order to keep the stream open at the time the RST_STREAM is sent.

The server side stream termination can be set in the same way, but under the server-response node.

The available options for error-code are:

NO_ERROR
PROTOCOL_ERROR
INTERNAL_ERROR
FLOW_CONTROL_ERROR
SETTINGS_TIMEOUT
STREAM_CLOSED
FRAME_SIZE_ERROR
REFUSED_STREAM
CANCEL
COMPRESSION_ERROR
CONNECT_ERROR
ENHANCE_YOUR_CALM
INADEQUATE_SECURITY
HTTP_1_1_REQUIRED

Finer grained frame ordering can also be specified via a delay node which specifies the time delay before sending the RST_STREAM frame. For example, to delay sending a RST_STREAM frame for 1 second, specify a delay: 1s directive like so:

      - RST_STREAM:
          error-code: INTERNAL_ERROR
          delay: 1s

A detailed description of the delay node can be found here.

GOAWAY frame

The GOAWAY frame acts similar to the RST_STREAM frame shown above. However, rather than terminating a stream, GOAWAY frame terminates the connection. It supports error-code and delay as described for RST_STREAM frame above.

HTTP/2 sessions also have a close-on-goaway directive. This boolean configuration informs how the Verifier client should behave when there are more streams configured in the replay YAML file after a GOAWAY frame is received (see RFC 7540 section 6.8 for details about the GOAWAY frame). When set to true, on receipt of a GOAWAY frame, the client terminates the connection after processing the streams that are currently being processed. When set to false, the client does not terminate the connection and continues with subsequent streams specified in the replay file. The default value of this option is true since that is in keeping with the RFC. Here's an example session configured with close-on-goaway set to false:

sessions:
- protocol:
  - name: http
    version: 2
  - name: tls
    sni: test_sni
  - name: tcp
  - name: ip
    version: 4
  close-on-goaway: false
  transactions:

Await

By protocol specification, HTTP/1 transactions are serialized. That is, apart from the rarely used pipelined request protocol feature (which is not supported by Proxy Verifier), each HTTP/1 transaction in a connection is not exchanged until the request and response of the previous one is completed. By contrast, the HTTP/2 protocol specifies that streams (i.e., HTTP/2 transactions) are multiplexed within their sessions (i.e., HTTP/2 connections). For the Verifier client, this means that, by default, requests for streams within a session are each sent immediately in the order that they are specified in the replay file without waiting for their respective responses. That is, if a replay file specifies three streams within a session, then the client will send the request for each stream as quickly as it can serialize it on the socket for the session before waiting for any responses.

The timing for sending the stream requests can be modified from this default behavior in several ways. As with replaying HTTP/1 traffic, if the streams contain timing specification nodes, then the client can be configured to replay the streams at a rate scaled to that timing information. See the documentation for the --rate argument for a description of how this works. Also, the user can include delay nodes to space out the replay of each stream.

In addition to these configurations supported in HTTP/1 traffic replay, for HTTP/2 and HTTP/3 the Verifier client supports the await node to control when it starts streams. This node takes a transaction key or a sequence of transaction keys, the responses of which are dependencies for sending the request for the associated stream. That is, a stream with an await node will not be sent until all the responses are received for the listed keys.

As an example, consider the specification of the following session with two streams:

# Specify an HTTP/2 session
- protocol:
  - name: http
    version: 2
  - name: tls
    sni: test_sni
  - name: tcp
  - name: ip

  transactions:

  # Stream 1
  - client-request:
      headers:
        fields:
        - [ :method, GET ]
        - [ :scheme, https ]
        - [ :authority, www.example.com ]
        - [ :path, /pictures/flower.jpeg ]
        - [ Content-Type, image/jpeg ]
        - [ uuid, first-stream ]

    server-response:
      headers:
        fields:
        - [ :status, 200 ]
        - [ Content-Type, image/jpeg ]
        - [ X-Response, first-response ]
      content:
        size: 3432

  # Stream 2
  - client-request:

      # This await will cause the Verifier client to hold off on sending this
      # request until the response to first-stream is received.
      await: first-stream

      headers:
        fields:
        - [ :method, GET ]
        - [ :scheme, https ]
        - [ :authority, www.example.com ]
        - [ :path, /pictures/flower.jpeg ]
        - [ Content-Type, image/jpeg ]
        - [ uuid, second-stream ]

    server-response:
      headers:
        fields:
        - [ :status, 200 ]
        - [ Content-Type, image/jpeg ]
        - [ X-Response, second-response ]
      content:
        size: 3432

Notice that the second transaction with key second-stream contains an await node. Without this node, the Verifier client would send both requests back to back. That is, the second-stream request would be sent immediately after the first-stream request was sent. With the await node specifying a dependency upon the first-stream transaction, however, the Verifier client will instead delay sending the second-stream request until the response to first-stream was received from the proxy.

There are a few things related to await nodes to be cognizant of:

await nodes can be used in conjunction with delay nodes for client-request specifications. When both are used, the Verifier client will first hold off on sending the request until all dependant responses specified by the keys in await are processed. Once the dependant responses are received, then the delay node is processed and the request is further delayed for the amount of time specified by the value of the delay node. Then the request is sent after the delay.
As stated before, the Verifier client is designed to processes streams and their directives in the order that they are specified in the transactions sequence for their stream. This means that any await or delay nodes for one transaction causes a corresponding delay for later nodes before they are processed. That is to say, if a session is specified with five streams, and the third stream has an await specified for the first two streams, then the fourth and fifth streams will also be delayed behind the third stream while it awaits the responses for the first and second streams, even if streams four and five contain no await nor delay nodes themselves.
Transaction keys specified via await can only be for earlier transactions in the same session that the client-request node is in. That is, the Verifier client does not support a stream in one session to await the response to a stream in a different session.

Protocol Specification

The above discussed the replay YAML nodes that describe how Proxy Verifier will craft HTTP layer traffic. This section discusses how the user specifies the lower layer protocols used to transport this HTTP traffic.

As stated above, each HTTP session is described as an item under the sessions node sequence. Each session takes a map. HTTP transactions are described under the transactions key described above. In addition to transactions, a session also takes an optional protocol node. This node takes an ordered sequence of maps, where each item in the sequence describes the characteristics of a protocol layer. The sequence is expected to be ordered from higher layer protocols (such as HTTP and TLS) to lower layer protocols (such as IP).

Here is an example protocol node along with sessions and transactions nodes provided to give some context:

sessions:

- protocol:
  - name: http
    version: 2
  - name: tls
    sni: test_sni
  - name: tcp
  - name: ip

  transactions:
  # ...

Note again how the protocol node is under the sessions node which takes a sequence of sessions. This sample shows the start of a single session that, in this case, provides a protocol description via a protocol key. This same session also has a truncated set of transactions that will be specified under the transactions key. Looking further at the protocol node, observe that this session has four layers described for it: http, tls, tcp, and ip. The http node specifies that the session should use the HTTP/2 protocol. The tls node specifies that the client should use an SNI of "test_sni" in the TLS client hello handshake. Further, this should be transported over TCP on IP.

The following nodes are supported for protocol:

Name	Node	Supported Values	Description
http
	version	{1, 2}	Whether to use HTTP/1 or HTTP/2.
tls
	sni	string	The SNI to send in the TLS handshake.
	request-certificate	boolean	Whether the client or server should request a certificate from the proxy.
	proxy-provided-certificate	boolean	This directs the same behavior as the request-certificate directive. This alias is helpful when the node describes what happened in the past, such as in the context of a replay file specified by Traffic Dump.
	verify-mode	{0-15}	The value to pass directly to OpenSSL's `SSL_set_verify` to control peer verification in the TLS handshake. This allows fine grained control over TLS verification behavior. `0` corresponds with SSL_VERIFY_NONE, `1` corresponds with SSL_VERIFY_PEER, `2` corresponds with SSL_VERIFY_FAIL_IF_NO_PEER_CERT, `4` corresponds with SSL_VERIFY_CLIENT_ONCE, and `8` corresponds with SSL_VERIFY_POST_HANDSHAKE. Any bitwise OR'd value of these values can be provided. For details about their behavior, see OpenSSL's SSL_verify_cb documentation.
	alpn-protocols	sequence of strings	This specifies the server's protocol list used in ALPN selection. See OpenSSL's SSL_select_next_proto documentation for details.
proxy-protocol
	version	{1, 2}	Whether to use PROXY header v1 or v2
	src-addr	string	The source address and port in the PROXY header. Specified in the format of `111.111.111.111:11111`.
	dst-addr	string	The destination address and port in the PROXY header. Specified in the same format of `src-addr`.
tcp
ip

The following protocol specification features are not currently implemented:

HTTP/2 is only supported over TLS. Proxy Verifier uses ALPN in the TLS handshake to negotiate HTTP/2 with the proxy. HTTP/2 upgrade from HTTP/1 without TLS is not supported.
The user cannot supply a TLS version to negotiate for the handshake. Currently, if the TLS node is present, Proxy Verifier will use the highest TLS version it can negotiate with the peer. This is OpenSSL's default behavior. An enhancement request to support A TLS version specification feature request is recorded in issue 101.
Similarly, the user cannot specify whether to use IPv4 or IPv6 via an ip:version node. Proxy Verifier can test IPv6, but it does so via the user passing IPv6 addresses on the commandline. An IP version feature request is recorded in issue 100.
Only TCP is supported. There have been recent discussions about adding HTTP/3 support, which is over UDP, but work for that has not yet started.
The PROXY protocol support is limited to the PROXY command via TCP over IPv4 or IPv6. Therefore,
- No support for AF_UNIX socket address as either source and destination address.
- No support for UDP transport type.
- No support for LOCAL command(in v2 header).

If there is no protocol node specified, then Proxy Verifier will default to establishing an HTTP/1 connection over TCP (no TLS).

PROXY protocol support

Generally speaking, a server sitting downstream from a proxy does not have visibility into the client's network socket information that lies behind the proxy. PROXY Protocol is a mechanism to provide visibility for this. PROXY protocol is a network protocol that communicates a client's source and destination IP and port information via a set of bytes at the start of a TCP connection from a proxy. Here is a link to the protocol description:

https://github.com/haproxy/haproxy/blob/master/doc/proxy-protocol.txt

Proxy Verifier supports sending and receiving the PROXY protocol, which can be helpful to verify the PROXY protocol behavior of the proxy under test. The feature can be enabled by specifying the proxy-protocol protocol node as outlined above. If enabled, the Verifier client would send out the PROXY protocol header at the beginning of the connection. Upon receiving a PROXY protocol header, the Verifier server would display it in the human-readable v1 format. Note that the specification of source and destination addresses are supported but not required; if not specified, the Proxy Verifier client would send out PROXY message with the source and destination addresses matching the underlying socket, similar to how curl --haproxy-protocol behaves.

Session and Transaction Delay Specification

A user can also stipulate per session and/or per transaction delays to be inserted by the Verifier client and server during the replay of traffic. This is done via the delay node which takes a unit-specified duration for the associated delay. During traffic replay, the delay is inserted before the associated session is established or before the client request or server response is sent. Proxy Verifier recognizes the following time duration units for the delay node:

Unit Suffix	Meaning
s	seconds
ms	milliseconds
us	microseconds

Here is a sample replay file snippet that demonstrates the specification of a set of delays:

sessions:
- delay: 2s

  transactions:

    client-request:
      delay: 15ms

      method: POST
      url: /a/path.jpeg
      version: '1.1'
      headers:
        fields:
        - [ Content-Length, '399' ]
        - [ Content-Type, image/jpeg ]
        - [ Host, example.com ]
        - [ uuid, 1 ]

  server-response:
    delay: 17000 us

    status: 200
    reason: OK
    headers:
      fields:
      - [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
      - [ Content-Type, image/jpeg ]
      - [ Transfer-Encoding, chunked ]
      - [ Connection, keep-alive ]
    content:
      size: 3432

Note that this example specifies the following delays:

The client delays 2 seconds before establishing the session.
The client also delays 15 milliseconds before sending the client request.
The server delays 17 milliseconds (17,000 microseconds) before sending the corresponding response after receiving the request.

Be aware of the following characteristics of the delay node:

The Verifier client interprets and implements delay for sessions in the sessions node and for transactions in the client-request node. The Verifier server interprets delay only for transactions in the server-response node and ignores sessions delays. Since the server is passive in receiving connections, it's not obvious what a server-side session delay would mean in this context.
Notice that Proxy Verifier supports microsecond level delay granularity, and does indeed faithfully insert delays at the appropriate times during replay with that precision of time. Be aware, however, that for the vast majority of networks anything more precise than a millisecond will not generally be useful.

See also --rate <requests/second> below for rate specification of transactions.

Keep Connection Open

In certain special situations, a user might need to keep the connection open after the final transaction in a session is done. To specify how long the connection needs to be kept open, the user can specify the duration as follows (value format is the same as Session and Transaction Delay Specification):

sessions:
- protocol:
  - name: http
    version: 2
  - name: tls
    sni: test_sni
  - name: tcp
  - name: ip
    version: 4

  keep-connection-open: 2s

  transactions:

    client-request:
      delay: 15ms

      method: POST
      url: /a/path.jpeg
      version: '1.1'
      headers:
        fields:
        - [ Content-Length, '399' ]
        - [ Content-Type, image/jpeg ]
        - [ Host, example.com ]
        - [ uuid, 1 ]

  server-response:
    delay: 17000 us

    status: 200
    reason: OK
    headers:
      fields:
      - [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
      - [ Content-Type, image/jpeg ]
      - [ Transfer-Encoding, chunked ]
      - [ Connection, keep-alive ]
    content:
      size: 3432

Traffic Verification Specification

In addition to replaying HTTP traffic as described above, Proxy Verifier also implements proxy traffic verification. For any given transaction, Proxy Verifier can optionally verify characteristics of an HTTP request line (for HTTP/1 requests - HTTP/2 requests do not have a request line), the response status, and the content of HTTP fields for requests and responses. Proxy Verifier also supports field verification of HTTP/2 pseudo header fields. Whereas client-request and server-response nodes direct the Verifier client and server (respectively) on how to send traffic, proxy-request and proxy-response nodes direct the server and client (respectively) on how to verify the traffic it receives from the proxy. Thus:

client-request nodes are used by the Verifier client to direct how it will generate requests it will send to the proxy.
server-response nodes are used by the Verifier server to direct how it will generate responses to send to the proxy.
proxy-request nodes are used by the Verifier server to direct how it will verify requests received from the proxy.
proxy-response nodes are used by the Verifier client to direct how it will verify responses received from the proxy.

In the event that the proxy does not produce the stipulated traffic according to a verification directive, the Proxy Verifier client or server that detects the violation will emit a log indicating the violation and will, upon process exit, return a non-zero status to the shell.

Traffic verification is an optional feature. Thus if Proxy Verifier is being used simply to replay traffic and the verification features are not helpful, then the user can simply omit the proxy-request and proxy-response nodes and no verification will be performed.

The following sections describe how to specify traffic verification in the YAML replay file.

Field Verification

Recall that in client-request and server-response nodes, fields are specified via a sequence of two items: the field name followed by the field value. For instance, the following incomplete client-request node contains the specification of a single Content-Length field (it is incomplete because it does not specify the method, request target, etc., but this snippet sufficiently demonstrates a typical description of an HTTP field):

  client-request:
    headers:
      fields:
      - [ Content-Length, 399 ]

For this client request, Proxy Verifier is directed to create a Content-Length HTTP field with a value of 399.

Field verification is specified in a similar manner, but the second item in the sequence is a map describing how the field should be verified. The map takes a value item describing the characteristics of the value to verify, if applicable, and an as item providing a directive describing how the field should be verified. Here is an example of a proxy-request node directing the Verifier server to verify the characteristics of the Content-Length field received from the proxy:

  proxy-request:
    headers:
      fields:
      - [ Content-Length, { value: 399, as: equal } ]

Observe the following from this verification example:

As described above, notice that the second entry in the field specification is a map instead of a scalar field value. The Proxy Verifier parser recognizes this map type as providing a verification specification.
As with client-request and server-response field specifications, the first item in the list describes the field name, in this case "Content-Length". This indicates that this verification rule applies to the "Content-Length" HTTP field from the proxy.
The directive is specified via the as key. In this example, equal is the directive for this particular field verification, indicating that the Verifier server should verify that the proxy's request for this transaction contains a Content-Length field with the exact value of 399.

Proxy Verifier supports six field verification directives:

Directive	Description
absent	The absence of a field with the specified field name.
present	The presence of a field with the specified field name and having any or no value.
equal	The presence of a field with the specified field name and a value equal to the specified value.
contains	The presence of a field with the specified name with a value containing the specified value.
prefix	The presence of a field with the specified name with a value prefixed with the specified value.
suffix	The presence of a field with the specified name with a value suffixed with the specified value.
includes	Helpful for multi-value fields. Specifies that the set of value strings exist in the header field values for a particular header name in any order. Note that each value is matched against the set like `contains`, so each value is a substring match.

For all of these field verification behaviors, field names are matched case insensitively while field values are matched case sensitively.

Thus the following field specification requests no field verification because it does not include a directive and the second item in the sequence is a scarlar:

  - [ X-Forwarded-For, 10.10.10.2 ]

Such non-operative fields can also be specified using the map syntax without an as item:

  - [ X-Forwarded-For, { value: 10.10.10.2 } ]

Fields like this in proxy-request and proxy-response nodes are permissible by Proxy Verifier's parser even though they have no functional impact (i.e., they do not direct Proxy Verifier's traffic behavior because they are not in client-request nor server-response nodes, and they do not describe any verification behavior). Allowing such fields affords the user the ability to record the proxy's traffic behavior in situations where field verification is not required or desired. For example, the Traffic Dump Traffic Server plugin records HTTP traffic and uses these proxy HTTP fields to indicate what fields were sent by the Traffic Server proxy to the client and the server. This can be helpful for analyzing the proxy's behavior via these replay files. Thus this proxy traffic recording function can be helpful to the user even though Proxy Verifier treats the fields as non-operable.

The following demonstrates the absent directive which specifies that the HTTP field X-Forwarded-For with any value should not have been sent by the proxy:

  - [ X-Forwarded-For, { as: absent } ]

The following demonstrates the present directive which specifies that the HTTP field X-Forwarded-For with any value should have been sent by the proxy:

  - [ X-Forwarded-For, { as: present } ]

Notice that for both the absent and the present directives, the value map item is not relevant and need not be provided and, in fact, will be ignored by Proxy Verifier if it is provided.

The following demonstrates the equal directive which specifies that X-Forwarded-For should have been received from the proxy with the exact value "10.10.10.2":

  - [ X-Forwarded-For, { value: 10.10.10.2, as: equal } ]

The following demonstrates the contains directive which specifies that X-Forwarded-For should have been received from the proxy containing the value "10" at any position in the field value:

  - [ X-Forwarded-For, { value: 10, as: contains } ]

The following demonstrates the prefix directive which specifies that X-Forwarded-For should have been received from the proxy with a field value starting with "1":

  - [ X-Forwarded-For, { value: 1, as: prefix } ]

The following demonstrates the suffix directive which specifies that X-Forwarded-For should have been received from the proxy with a field value ending with "2":

  - [ X-Forwarded-For, { value: 2, as: suffix } ]

The following demonstrates the includes directive which specifies that Set-Cookie should have been received from the proxy including the values "A" and "B" at any position in the field value:

  - [ Set-Cookie, { value: [A, B] , as: includes } ]

Proxy Verifier supports inverting the result of any rule by using not instead of as. The following demonstrates the prefix directive which specifies that X-Forwarded-For should have been received from the proxy with a field value not starting with "a":

  - [ X-Forwarded-For, { value: a, not: prefix } ]

Proxy Verifier also supports ignoring the upper/lower case distinction with another directive: case: ignore. The following demonstrates the suffix directive which specifies that X-Forwarded-For should have been received from the proxy with a field value starting with "a" or "A":

  - [ X-Forwarded-For, { value: a, as: prefix, case: ignore } ]

The not and case: ignore directives can both be applied on the same rule. The following demonstrates the suffix directive which specifies that X-Forwarded-For should have been received from the proxy with a field value not starting with "a" nor "A":

  - [ X-Forwarded-For, { value: a, not: prefix, case: ignore } ]

In addition to HTTP/2 trailer header replay discussed above, Proxy Verifier also supports trailer header verification, as demonstrated below:

proxy-response:
  # Other verifications...
  trailers:
    fields:
    # Verify the client receives the response trailers.
    - [ x-test-trailer-1, { value: one, as: equal } ]
    - [ x-test-trailer-2, { value: two, as: equal } ]

To perform multi-value field verification, a specific format must be adhered to. This format involves specifying each value within a sequence, ensuring that each value is individually verified according to the defined rules.

Be aware that field verification is order sensitive. That is, the field values will be verified in the order that they are specified in the verification rule.

See example below, value B1 and B2 are specified in the same order in the header fields and verification rule:

server-response:
  headers:
    fields:
    - [:status, 200]
    - [Content-Type, text/html]
    - [Content-Length, '11']
    - [Set-Cookie, "A1=111"]
    - [Set-Cookie, "A2=222"]
    - [Set-Cookie, "B1=333"]
  content:
    encoding: plain
    data: server_test
    size: 11

proxy-response:
  headers:
    fields:
    - [Set-Cookie, { value: [A1=111, A2=222, B1=333] , as: equal }]

There is an exception for the includes check, where it is not order sensitive. See exammple below:

server-response:
  headers:
    fields:
    - [:status, 200]
    - [Content-Type, text/html]
    - [Content-Length, '11']
    - [Set-Cookie, "A1=111"]
    - [Set-Cookie, "A2=222"]
    - [Set-Cookie, "B1=333"]
    - [Set-Cookie, "B2=444"]
    - [Set-Cookie, "C1=555"]
    - [Set-Cookie, "C2=666"]
    - [Set-Cookie, "D1=777"]
    - [Set-Cookie, "D2=888"]
  content:
    encoding: plain
    data: server_test
    size: 11

proxy-response:
  headers:
    fields:
    - [Set-Cookie, { value: [B2=, A2=, C2=, D1=, C1=] , as: includes }]

URL Verification

In a manner similar to field verification described above, a mechanism exists to verify the parts of URLs in the request line being received from the proxy by the server. This mechanism is useful for verifying the request targets of HTTP/1 requests. For HTTP/2 requests, the analogous verification is done via pseudo header field verification of the :scheme, :authority, and :path fields using the above described field verification.

Request target verification rules are stipulated via a sequence value for the url node rather than the scalar value used in client-request nodes. The verifiable parts of the request target follow the URI specification (see RFC 3986 section 3 for the formal definition of these various terms):

scheme
host
port
authority (also known as net-loc, the combination of host and port),
path
query
fragment

For example, consider the following request line:

    GET http://example.one:8080/path?query=q#Frag HTTP/1.1

The request URL in this case is case is http://example.one:8080/path?query=q#Frag. The Verifier server can be configured to verify the various parts of such URLs using the same map sytax explained above for field verification using any of those same directives (equal, contains, etc.). Continuing with this example URL, the following uses the equal directive for each part of the URL, thus verifying that the request URL exactly matches the URL given in this example and emitting a verification error message if any parts of the URL do not match:

  proxy-request:
    url:
    - [ scheme,   { value: http,        as: equal } ]
    - [ host,     { value: example.one, as: equal } ]
    - [ port,     { value: 8080,        as: equal } ]
    - [ path,     { value: /path,       as: equal } ]
    - [ query,    { value: query=q,     as: equal } ]
    - [ fragment, { value: Frag,        as: equal } ]

Alternatively to host and port, authority, with an alias of net-loc, is supported, which is the combination of the two:

  - [ authority, { value: example.one:8080, as: equal } ]

As another example using other directives, consider a request URL of /path/x/y?query=q#Frag. Verification of this can be specified with the following:

  proxy-request:
    url:
    - [ scheme,    { value: http,      as: absent } ]
    - [ authority, { value: foo,       as: absent } ]
    - [ path,      { value: /path/x/y, as: equal } ]
    - [ query,     { value: query=q,   as: equal } ]
    - [ fragment,  { value: foo,       as: present } ]

Note that scheme and authority parts both use absent directives because this particular URL just has path, query, and fragment components. Thus, with this verification specification, if the proxy includes scheme or authority in the request target, it will result in a verification failure. The path and query components must match /path/x/y and query=q exactly because they use the equal directive. The fragment of the URL is verified with the present directive in this case, indicating that the received URL from the proxy only needs to have some fragment of any value to pass this specified verification.

Status Verification

Proxy HTTP response status verification is specified in proxy-response nodes. In this case, the response status that the Verifier client should expect from the proxy is specified in the same way that directs the Verifier server in what response status should be sent for a given request. Both response code, such as "403", and HTTP/1 response reason string, such as "Forbidden", are supported for verification.

For example, the following complete proxy-response node directs the Proxy Verifier client to verify that the proxy replies to the associated HTTP request with a 404 status:

  proxy-response:
    status: 404
    reason: Not Found

This verification directive applies to HTTP/1 transactions. For HTTP/2, status verification is specified via :status pseudo header field verification using the field verification mechanism described above.

Body Verification

In a manner similar to field verification described above, a mechanism exists to verify the body content of a request or response. To specify a rule for verifying body content, a new verify node should be added under the content node. The rules follow the same map syntax as described for field verification.

  proxy-request:
    content:
      verify: {value: test1, as: equal}

  proxy-response:
    content:
      verify: {value: test2, as: contains}

The value node in the verify node can be ommited if the data node is used to specify the content, since body content can get very long, and/or multi-lined. However, value node has priority over the data node, meaning if there is a value node, then the data will be ignored.

  proxy-request:
    content:
      encoding: plain
      data: test1
      verify: {as: equal}

  proxy-response:
    content:
      encoding: plain
      data: test2
      verify: {as: contains}

Note that only one body verification node is needed, even if multiple DATA frames are specified as described in HEADERS and DATA frame. The verification node need to combine all the values specified for the DATA frames. See the example below:

  client-request:
    frames:
    - HEADERS:
        headers:
          fields:
          - [:method, POST]
          - [:scheme, https]
          - [:authority, example.data.com]
          - [:path, /a/path]
          - [Content-Type, text/html]
          - [uuid, 1]
    - DATA:
          content:
          encoding: plain
          data: client_data_1
    - DATA:
        content:
          encoding: plain
          data: client_data_2

  proxy-request:
    content:
      encoding: plain
      data: client_data_1client_data_2
      verify: {as: equal}

Example Replay File

The sections leading up to this one have described each of the major components of a YAML Proxy Verifier replay file. Putting these components together, the following complete sample replay file demonstrates the description of the replay of two sessions: one an HTTP/1.1 session, the second an HTTP/2 session. Each session contains a single transaction with verification of certain parts of the HTTP messages.

meta:
    version: '1.0'

sessions:

#
# First session: since there is no "protocol" node for this session,
# HTTP/1.1 over TCP (no TLS) is assumed.
#
- transactions:

    #
    # Direct the Proxy Verifier client to send a POST request with a body of
    # 399 bytes.
    #
  - client-request:
      method: POST
      url: /pictures/flower.jpeg
      version: '1.1'
      headers:
        fields:
        - [ Host, www.example.com ]
        - [ Content-Type, image/jpeg ]
        - [ Content-Length, '399' ]
        - [ uuid, first-request ]
      # A "content" node is not needed if a Content-Length field is specified.

    #
    # Direct the Proxy Verifier server to verify that the request received from
    # the proxy has a path in the request target that contains "flower.jpeg",
    # has a path that is not prefixed with "JPEG" (case insensitively),
    # and has the Content-Length field of any value.
    #
    proxy-request:
      url:
      - [ path, { value: flower.jpeg, as: contains } ]
      - [ path, { value: JPEG, not: prefix, case: ignore } ]

      headers:
        fields:
        - [ Content-Length, { value: '399', as: present } ]

    #
    # Direct the Proxy Verifier server to reply with a 200 OK response with a body
    # of 3,432 bytes.
    #
    server-response:
        status: 200
        reason: OK
        headers:
          fields:
          - [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
          - [ Content-Type, image/jpeg ]
          - [ Transfer-Encoding, chunked ]
          - [ Connection, keep-alive ]
        # Unlike the request which contains a Content-Length, this response
        # will require a "content" node to specify the size of the body.
        # Otherwise Proxy Verifier has no way of knowing how large the response
        # should be.
        content:
          size: 3432

    #
    # Direct the Proxy Verifier client to verify that it receives a 200 OK from
    # the proxy with a `Transfer-Encoding: chunked` header field.
    #
    proxy-response:
      status: 200
      headers:
        fields:
        - [ Transfer-Encoding, { value: chunked, as: equal } ]

#
# For the second session, we use a protocol node to configure HTTP/2 using an
# SNI of # test_sni in the TLS handshake.
#
- protocol:
  - name: http
    version: 2
  - name: tls
    sni: test_sni
  - name: tcp
  - name: ip

  transactions:

  #
  # Direct the Proxy Verifier client to send a POST request with a body of
  # 399 bytes.
  #
  - client-request:
      headers:
        fields:
        - [ :method, POST ]
        - [ :scheme, https ]
        - [ :authority, www.example.com ]
        - [ :path, /pictures/flower.jpeg ]
        - [ Content-Type, image/jpeg ]
        - [ uuid, second-request ]
      content:
        size: 399

    #
    # Direct the Proxy Verifier server to verify that the request received from
    # the proxy has a path pseudo header field that contains "flower.jpeg"
    # and has a field "Content-Type: image/jpeg".
    #
    proxy-request:
      url:
      - [ path, { value: flower.jpeg, as: contains } ]

      headers:
        fields:
        - [ :method, POST ]
        - [ :scheme, https ]
        - [ :authority, www.example.com ]
        - [ :path,        { value: flower.jpeg, as: contains } ]
        - [ Content-Type, { value: image/jpeg,  as: equal } ]

    #
    # Direct the Proxy Verifier server to reply with a 200 OK response with a body
    # of 3,432 bytes.
    #
    server-response:
      headers:
        fields:
        - [ :status, 200 ]
        - [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
        - [ Content-Type, image/jpeg ]
      content:
        size: 3432

    #
    # Direct the Proxy Verifier client to verify that it receives a 200 OK from
    # the proxy.
    #
    proxy-response:
      status: 200

#
# For the third session, we demonstrate how body verification should be specified.
#
- protocol:
  - name: http
    version: 1.1
  - name: tls
    sni: test_sni
  - name: tcp
  - name: ip
    version: 4

  transactions:

  #
  # Direct the Proxy Verifier client to send a POST request with a body of
  # 11 bytes.
  #
  - client-request:
      method: POST
      url: /a/path
      version: '1.1'
      headers:
        fields:
        - [ Host, example.data.com ]
        - [ Content-Type, text/html ]
        - [ Content-Length, '11' ]
        - [ uuid, third-request ]
      content:
        encoding: plain
        data: client_test
        size: 11

    #
    # Direct the Proxy Verifier server to verify that the request received from
    # the proxy has body content "client_test".
    #
    proxy-request:
      content:
        verify: { value: client_test, as: equal }

    #
    # Direct the Proxy Verifier server to reply with a 200 OK response with a body
    # of 11 bytes.
    #
    server-response:
      status: 200
      reason: OK
      headers:
        fields:
        - [ Content-Type, text/html ]
        - [ Content-Length, '11' ]
      content:
        encoding: plain
        data: |-
          server
          test
        size: 11

    #
    # Direct the Proxy Verifier client to verify that the response received from
    # the proxy has body content "server\ntest".
    #
    proxy-response:
      content:
        encoding: plain
        data: |-
          server
          test
        verify: { as: equal }

Installing

Prebuilt Binaries

Starting with the v2.2.0 release, statically linked binaries for Linux and Mac are provided with the release in the Releases page. If you do not need your own customized build of Proxy Verifier, the easiest way to start using it is to simply download the proxy-verifier tar.gz for the desired release, untar it on the desired box, and copy the verifier-client and verifier-server binaries to a convenient location from which to run them. The Linux binaries should run on Ubuntu, Alma/CentOS/Fedora/RHEL, FreeBSD, and other Linux flavors.

Building from Source

These instructions describe how to build a copy of the Proxy Verifier project on your local machine for development and testing purposes.

Prerequisites

Proxy Verifier is built using SCons. Scons is a Python module, so installing it is as straightforward as installing any Python package. A top-level Pipfile is provided to install Scons and its use is described and assumed in these instructions, but it can also be installed using pip if preferred.

Scons will clone and build the dependent libraries using Automake. Thus building will require the installation of the following system packages:

git
pipenv
autoconf
libtool
pkg-config

For system-specific commands to install these packages (Ubuntu, CentOS, etc.), one can view the Dockerfile documents provided under docker. These demonstrate, for each system, what commands are used to install these package dependencies. Naturally, performing a docker build against these Dockerfiles can also be used to create Docker images, containers from which builds can be performed.

In addition to the above system package dependencies, Proxy Verifier utilizes the following C++ libraries:

OpenSSL is used to implement TLS encryption. Proxy Verifier requires the version of OpenSSL that supports QUIC.
Nghttp2 is used for parsing HTTP/2 traffic.
ngtcp2 is used for parsing QUIC traffic.
nghttp3 is used for parsing HTTP/3 traffic.
yaml-cpp is used for parsing the YAML replay files.
libswoc are a set of C++ library extensions to support string parsing, memory management, logging, and other features.

Note: None of these libraries need to be explicitly installed before you build. By default, Scons will fetch and build each of these libraries as a part of building the project.

Build

Once the above-listed system packages (git, autoconf, etc.) are installed on your system, you can build Proxy Verifier using Scons. This involves first creating the Python virtual environment and then running the scons command to build Proxy Verifier. Here is an example invocation:

# Install scons and any of its Python requirements. This only needs to be
# done once before the first invocation of scons.
#
# Note: for older RHEL/CentOS systems, you will have to souce the appropriate
# Python 3 enable script to initialize the correct Python 3 environment. For
# example:
# source /opt/rh/rh-python38/enable
pipenv install

# Now run scons to build proxy-verifier.
pipenv run scons -j4

This will build and install verifier-client and verifier-server in the bin/ directory at the root of the repository. -j4 directs Scons to build with 4 threads. Adjust according to the capabilities of your build system.

Using Prebuilt Libraries

As mentioned above, Scons will by default fetch the various library dependencies (OpenSSL, Nghttp2, etc.), build, and manage those for you. If you do not change the fetched source code for these libraries, they will not be rebuilt after the first scons build invocation. This behavior is convenient as it relieves the burden of fetching and building these libraries from the developer. However, Scons will rescan the fetched source trees for these libraries on every call of scons to inspect them for any changes. For long-term development projects, a developer may find it more efficient to build these libraries externally and relieve Scons from managing them. To conveniently support this, the build_library_dependencies.sh script is provided to build these libraries. To build and install the libraries, run that script, passing as an argument the desired install location for the various libraries. Then point Scons to those libraries using various --with directives.

Here's an example invocation of scons along with the use of the library build script:

# Alter this to your desired library location.
http3_libs_dir=${HOME}/src/http3_libs

bash ./tools/build_http3_dependencies.sh ${http3_libs_dir}

pipenv install
pipenv run scons \
    -j4 \
    --with-ssl=${http3_libs_dir}/openssl \
    --with-nghttp2=${http3_libs_dir}/nghttp2 \
    --with-ngtcp2=${http3_libs_dir}/ngtcp2 \
    --with-nghttp3=${http3_libs_dir}/nghttp3

The Dockerfile documents run this build script, installing the HTTP packages in /opt. Therefore, if you are developing in a container made from images generated from these Dockerfile documents, you can use the following scons command to build Proxy Verifier:

pipenv install
pipenv run scons \
    -j4 \
    --with-ssl=/opt/openssl \
    --with-nghttp2=/opt/nghttp2 \
    --with-ngtcp2=/opt/ngtcp2 \
    --with-nghttp3=/opt/nghttp3

As a further convenience, if these libraries (openssl, nghttp2, ngtcp2, and nghttp3, with those exact names) exist under a single directory, such as is the case with images built from the provided Dockerfile documemnts, then you can specify the location of these libraries with a single --with-libs argument. Thus the previous command can be expressed like so:

pipenv install
pipenv run scons -j4 --with-libs=/opt

ASan Instrumentation

The local Sconstruct file is configured to take an optional --enable-asan parameter. If this is passed to the scons build line then the Proxy Verifier objects and binaries will be compiled and linked with the flags that instrument them for AddressSanatizer. This assumes that the system has the AddressSanatizer library installed on the system. Thus the above invocation would look like the following to compile it with AddressSanitizer instrumentation:

pipenv install
pipenv run scons \
    -j4 \
    --with-ssl=/path/to/openssl \
    --with-nghttp2=/path/to/nghttp2 \
    --with-ngtcp2=/path/to/ngtcp2 \
    --with-nghttp3=/path/to/nghttp3 \
    --enable-asan

Debug Build

By default, Scons will build the Proxy Verifier project in release mode. This means that the binaries will compiled with optimization. If an unoptimized debug build is desired, then pass the --cfg=debug option to scons:

pipenv run scons -j4 --cfg=debug

Statically Link

The current Scons configuration dynamically links the binaries with the various OpenSSL and HTTP build libraries. This is fine for local testing and execution, but can be inconvenient when copying the binaries to other machines. Ideally the Sconstruct file can be updated to support an option to link the binaries statically. This currently does not exist and is not trivial to create. Future updates to scons-parts may help with this. As a current stopgap measure, tools/build_static is provided which automatically links the Verifier binaries statically. It is run from the root directory of your repository like so:

./tools/build_static

By default this builds Proxy Verifier with the following invocation:

pipenv run scons -j$(nproc)

Any arguments passed to build_static will be passed through to the scons command. Thus, if you desire to build Proxy Verifier with --with-libs=/opt, run the script like so:

./tools/build_static --with-libs=/opt`

Running the Tests

Unit Tests

To build and run the unit tests, use the run_utest Scons target (this assumes you previously ran pipenv install, see above):

pipenv run scons \
    -j4 \
    --with-ssl=/path/to/openssl \
    --with-nghttp2=/path/to/nghttp2 \
    --with-ngtcp2=/path/to/ngtcp2 \
    --with-nghttp3=/path/to/nghttp3 \
    run_utest::

Gold Tests

Proxy Verifier ships with a set of automated end to end tests written using the AuTest framework. To run them, simply run the autest.sh script:

cd test/autests
./autest.sh

When doing development for which a particular AuTest is relevant, the -f option can be used to run just a particular test (or set of tests, specified in a space-separated list). For instance, the following invocation runs just the http and https tests:

./autest.sh -f http https

AuTest supports a variety of other options. Run ./autest.sh --help to get a quick description of the various command-line options. See the AuTest Documentation for further details about the framework.

A note for macOS: The Python virtual environment for these gold tests requires the cryptograpy package as a dependency of the pyOpenSSL package. Pipenv will install this automatically, but the installation of the cryptography package will require compiling certain c files against OpenSSL. macOS has its own SSL libraries which brew's version of OpenSSL does not replace, for understandable reasons. The building of cryptography will fail against the system's SSL libraries. To point the build to brew's OpenSSL libraries, the autest.sh script exports the following variables before running pipenv install:

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"

Thus if you stick with using the autest.sh script you do not need to worry about this. But if you install pipenv by hand rather than through the autest.sh script on macOS, then keep this in mind and export those variables before running pipenv install.

Usage

This section describes how to run the Proxy Verifier client and server at the command line.

Required Arguments

At a high level, Proxy Verifier is run in the following manner:

Run the verifier-server with the set of HTTP and HTTPS ports to listen on configured though the command line. The directory containing the replay file is also configured through a command line argument.
Configure and run the proxy to listen on a set of HTTP and HTTPS ports and to proxy those connections to the listening verifier-server ports.
Run the verifier-client with the sets of HTTP and HTTPS ports on which to connect configured though the command line. The directory containing the replay file is also configured through a command line argument.

Here's an example invocation of the verifier-server, configuring it to listen on localhost port 8080 for HTTP connections and localhost port 4443 for HTTPS connections:

verifier-server \
    run \
    --listen-http 127.0.0.1:8080 \
    --listen-https 127.0.0.1:4443 \
    --server-cert <server_cert> \
    --ca-certs <file_or_directory_of_ca_certs> \
    <replay_file_or_directory>

Here's an example invocation of the verifier-client, configuring it to connect to the proxy which has been configured to listen on localhost port 8081 for HTTP connections and localhost port 4444 for HTTPS connections:

verifier-client \
    run \
    --client-cert <client_cert> \
    --ca-certs <file_or_directory_of_ca_certs> \
    --connect-http 127.0.0.1:8081 \
    --connect-https 127.0.0.1:4444 \
    <replay_file_or_directory>

With these two invocations, the verifier-client and verifier-server will replay the sessions and transactions in <replay_file_or_directory> and perform any field verification described therein.

On the server either --listen-http or --listen-https or both must be provided. That is, for example, if you are only testing HTTPS traffic, you may only specify --listen-https. The same is true on the client: either --connect-http or --connect-https or both must be provided. These address arguments take a comma-separated list of address/port pairs to specify multiple listening or connecting sockets. The processing of these arguments automatically detects any IPv6 addresses if provided. The client's processing of --connect-http and --connect-https arguments will resolve fully qualified domain names.

Note that the --client-cert and --server-cert both take either a certificate file containing the public and private key or a directory containing pem and key files. Similarly, the --ca-certs takes either a file containing one or more certificates or a directory with separate certificate files. For convenience, the test/keys directory contains key files which can be used for testing. These certificate arguments are only required if HTTPS traffic will be replayed.

Optional Arguments

--format <format-specification>

Each transaction has to be uniquely identifiable by the client and server in a way that is consistent across both replay file parsing and traffic replay processing. Whatever attributes we use from the messages to uniquely identify transactions is called the "key" for the dataset. The ability to uniquely identify these messages is important for at least the following reasons:

When the Verifier server receives a request, it has to know from which of the set of parsed transactions it should generate a response. At the time of processing an incoming message, all it has to go on is the request header line and the request header fields. From these, it has to be able to identify which of the potentially thousands of parsed transactions from the replay input files it should generate a response.
When the client and server perform field verification, they need to know what particular verification rules specified in the replay files should be applied to the given incoming message.
If the client and server are processing many transactions, generic log messages could be near useless if there was not a way for the logs to identify individual transactions to the user somehow.

By default the Verifier client and server both expect a uuid header field value to function as the key.

If the user would like to use other attributes as a key, they can specify something else via the --format argument. The format argument currently supports generating a key on arbitrary field values and the URL of the request. Some example --format expressions include:

--format "{field.uuid}": This is the default key format. It treats the UUID header field value as the transaction key.
--format "{url}": Treat the request URL as the key.
--format "{field.host}": Treat the Host header field value as the key.
--format "{field.host}/{url}": Treat the combination of the Host header field and the request URL as the key.

--keys <key1 key2 ... keyn>

--keys can be passed to the verifier-client to specify a subset of keys from the replay file to run. Only the transactions from the space-separated list of keys will be replayed. For example, the following invocation will only run the transactions with keys whose values are 3 and 5:

verifier-client \
    run \
    <replay_file_diretory> \
    127.0.0.1:8082 \
    127.0.0.1:4443 \
    --keys 3 5

This is a client-side only option.

--verbose

Proxy Verifier has four levels of verbosity that it can run with:

Verbosity	Description
error	Transactions either failed to run or failed verification.
warning	A non-failing problem occurred but something is likely to go wrong in the future.
info	High level test execution information.
diag	Low level debug information.

Each level implies the ones above it. Thus, if a user specifies a verbosity level of warning, then both warning and error messages are reported.

By default, Proxy Verifier runs at info verbosity, only producing summary output by both the client and the server along with any warnings and errors it found. This can be tweaked via the --verbose flag. Here's an example of requesting the most verbose level of logging (diag):

verifier-client \
    run \
    <replay_file_diretory> \
    127.0.0.1:8082 \
    127.0.0.1:4443 \
    --verbose diag

--interface <interface>

Initiate connections from the specified interface, such as eth0:1.

This is a client-side only option.

--no-proxy

As explained above, replay files contain traffic information for both client to proxy traffic and proxy to server traffic. Under certain circumstances it may be helpful to run the Verifier client directly against the Verifier server. This can be useful while developing Proxy Verifier itself, for example, allowing the developer to do some limited testing without requiring the setup of a test proxy.

To support this, the Verifier client has the --no-proxy option. If this option is used, then the client has its expectations configured such that it assumes it is run against the Verifier server rather than a proxy. Effectively this means that instead of trying to run the client to proxy traffic, it will instead act as the proxy host for the Verifier server and will run the proxy to server traffic. Concretely, this means that the Verifier client will replay the proxy-request and proxy-response nodes rather than the client-request and client-response nodes.

This is a client-side only option.

--strict

Generally, very little about the replayed traffic is verified except what is explicitly specified via field verification (see above). This is by design, allowing the user to replay traffic with only the requested content being verified. In high-volume cases, such as situations where Proxy Verifier is being used to scale test the proxy, traffic verification may be considered unimportant or even unnecessarily noisy. If, however, the user wants every field to be verified regardless of specification, then the --strict option can be passed to either or both the Proxy Verifier client and server to report any verification issues against every field specified in the replay file.

--rate <requests/second>

By default, the client will replay the session and transactions in the replay files as fast as possible. If the user desires to configure the client to replay the transactions at a particular rate, they can provide the --rate argument. This argument takes the number of requests per second the client will attempt to send requests at.

Note session and transaction timing data can be specified in the replay files. These are provided via start-time nodes for each session and transaction. start-time takes as a value the number of nanoseconds since Unix epoch (or whatever other time of reference observed by all start-time nodes in the set of replay files being run) associated for that session or transaction. With this timing information, if --rate is provided, Proxy Verifier simply scales the relative time deltas between sessions and transactions that appropriately achieves the desired transaction rate. Traffic Dump records such timing information when it writes replay files. In the absence of start-time nodes, Proxy Verifier will attempt to apply an appropriate uniform delay across the sessions and transactions to achieve the specified --rate value.

This is a client-side only option.

--repeat <number>

By default, the client will replay all the transactions once in the set of input replay files. If the user would like the client to automatically repeat this set a number of times, they can provide the --repeat option. The argument takes the number of times the client should replay the entire dataset.

This is a client-side only option.

--run-continuously

Run the set of sessions in the input replay files in an infinite loop. Passing this option can be thought of as --repeat with a value of infinity.

This is a client-side only option.

--thread-limit <number>

Each connection, corresponding to a session in a replay file, is dispatched on the client in parallel. Likewise, each accepted connection on the server is handled in parallel. Each of these sessions is handled via a single thread of execution. By default, Proxy Verifier limits the number of threads for handling these connections to 2,000. This limit can be changed via the --thread-limit option. Setting a value of 1 on the client will effectively cause sessions to be replayed in serial.

--qlog-dir <directory>

Proxy Verifier supports logging of replayed QUIC traffic information conformant to the qlog format. If the --qlog-dir option is provided, then qlog files for all replayed QUIC traffic will be written into the specified directory. qlog diagnostic logging is disabled by default.

--tls-secrets-log-file <secrets_log_file_name>

To facilitate debugging, Proxy Verifier supports logging TLS keys for encrypted replayed traffic. If this option is used, TLS key logging will be appended to the specified filename. This file can then be provided to protocol analyzers such as Wireshark to decrypt the traffic. TLS key logging is disabled by default.

--send-buffer-size <size>

Configure a SO_SNDBUF value to set on the server socket. This can be a helpful parameter to tune when dealing with the replay of large response bodies. If this optional value is not set, then the SO_SNDBUF option is not set on the socket and, thus, the system default socket buffer size will be used.

This is a server-side only option.

--poll-timeout <timeout_ms>

When Proxy Verifier performs read and write operations, it does so using non-blocking sockets with a timeout. By default, this timeout is set to 5 seconds (5,000 milliseconds). This optional argument provides a way to specify a different timeout in milliseconds for these operations.

Tools

This section describes how to use some of the scripts under the tools directory.

Replay Gen replay_gen.py

This tool is used to generate mock replay files for easy testing. Listed below are the available arguments for this script.

-n,--number <NUMBER>

Number of total transactions.

-tl,--trans-lower <TRANS_LOWER>

The lower limit of transactions per session.

-tu,--trans-upper <TRANS_UPPER>

The upper limit of transactions per session.

-sl,--sess-lower <SESS_LOWER>

The lower limit of sessions per file.

-su,--sess-upper <SESS_UPPER>

The upper limit of sessions per file.

-tp,--trans-protocols <TRANS_PROTOCOLS>

A comma separated list of protocols that are allowed to be generated. Available options are: http, tls, h2, all.

-u,--url-file <URL_FILE>

Path to a file with the list of URLs that can be used. The URL list file can be acquired by running the Remap Config to URL List script described below.

-o,--output <OUTPUT>

Path to a directory where the replay files are generated.

-p,--prefix <PREFIX>

Prefix for the replay file names.

-j,--out-json

Dump replay files in JSON format. By default replay files will be formatted as YAML.

Remap Config to URL List remap_config_to_url_list.py

This tool converts a remap.config file to a URL list file that is used by Replay Gen Listed below are the available arguments for this script.

-o,--output <OUTPUT_FILE>

A filename to which to write the list of URLs. Defaults to stdout.

--no-ip

Ignore ip address (in the "replacement" section) in the remap.config file.

Contribute

Please refer to CONTRIBUTING for information about how to get involved. We welcome issues, questions, and pull requests.

License

This project is licensed under the terms of the Apache 2.0 open source license. Please refer to LICENSE for the full terms.

Files

README.md

Latest commit

History

README.md

File metadata and controls

Table of Contents

Proxy Verifier

Traffic Replay Specification

HTTP Specification

Server Response Lookup

HTTP/2 Specification

HEADERS and DATA frame

RST_STREAM frame

GOAWAY frame

Await

Protocol Specification

PROXY protocol support

Session and Transaction Delay Specification

Keep Connection Open

Traffic Verification Specification

Field Verification

URL Verification

Status Verification

Body Verification

Example Replay File

Installing

Prebuilt Binaries

Building from Source

Prerequisites

Build

Using Prebuilt Libraries

ASan Instrumentation

Debug Build

Statically Link

Running the Tests

Unit Tests

Gold Tests

Usage

Required Arguments

Optional Arguments

--format <format-specification>

--keys <key1 key2 ... keyn>

--verbose

--interface <interface>

--no-proxy

--strict

--rate <requests/second>

--repeat <number>

--run-continuously

--thread-limit <number>

--qlog-dir <directory>

--tls-secrets-log-file <secrets_log_file_name>

--send-buffer-size <size>

--poll-timeout <timeout_ms>

Tools

Replay Gen replay_gen.py

-n,--number <NUMBER>

-tl,--trans-lower <TRANS_LOWER>

-tu,--trans-upper <TRANS_UPPER>

-sl,--sess-lower <SESS_LOWER>

-su,--sess-upper <SESS_UPPER>

-tp,--trans-protocols <TRANS_PROTOCOLS>

-u,--url-file <URL_FILE>

-o,--output <OUTPUT>

-p,--prefix <PREFIX>

-j,--out-json

Remap Config to URL List remap_config_to_url_list.py

-o,--output <OUTPUT_FILE>

--no-ip

Contribute

License