The Reductionist API accepts HTTP POST requests to /v1/{operation}, where {operation} is the name of the operation to perform, one of count, min, max, sum or select.
+The request body should be a JSON object of the form:
{
+ // The URL for the S3 source
+ // - required
+ "source": "https://s3.example.com/,
+
+ // The name of the S3 bucket
+ // - required
+ "bucket": "my-bucket",
+
+ // The path to the object within the bucket
+ // - required
+ "object": "path/to/object",
+
+ // The data type to use when interpreting binary data
+ // - required
+ "dtype": "int32|int64|uint32|uint64|float32|float64",
+
+ // The byte order (endianness) of the data
+ // - optional, defaults to native byte order of Reductionist server
+ "byte_order": "big|little",
+
+ // The offset in bytes to use when reading data
+ // - optional, defaults to zero
+ "offset": 0,
+
+ // The number of bytes to read
+ // - optional, defaults to the size of the entire object
+ "size": 128,
+
+ // The shape of the data (i.e. the size of each dimension)
+ // - optional, defaults to a simple 1D array
+ "shape": [20, 5],
+
+ // Indicates whether the data is in C order (row major)
+ // or Fortran order (column major, indicated by 'F')
+ // - optional, defaults to 'C'
+ "order": "C|F",
+
+ // An array of [start, end, stride] tuples indicating the data to be operated on
+ // (if given, you must supply one tuple per element of "shape")
+ // - optional, defaults to the whole array
+ "selection": [
+ [0, 19, 2],
+ [1, 3, 1]
+ ],
+
+ // Algorithm used to compress the data
+ // - optional, defaults to no compression
+ "compression": {"id": "gzip|zlib"},
+
+ // List of algorithms used to filter the data
+ // - optional, defaults to no filters
+ "filters": [{"id": "shuffle", "element_size": 4}],
+
+ // Missing data description
+ // - optional, defaults to no missing data
+ // - exactly one of the keys below should be specified
+ // - the values should match the data type (dtype)
+ "missing": {
+ "missing_value": 42,
+ "missing_values": [42, -42],
+ "valid_min": 42,
+ "valid_max": 42,
+ "valid_range": [-42, 42],
+ }
+}
+Request authentication is implemented using Basic Auth with the username and password consisting of your S3 Access Key ID and Secret Access Key, respectively. +Unauthenticated access to S3 is possible by omitting the basic auth header.
+On success, all operations return HTTP 200 OK with the response using the same datatype as specified in the request except for count which always returns the result as int64.
+The server returns the following headers with the HTTP response:
x-activestorage-dtype: The data type of the data in the response payload. One of int32, int64, uint32, uint64, float32 or float64.x-activestorage-byte-order: The byte order of the data in the response payload. Either big or little.x-activestorage-shape: A JSON-encoded list of numbers describing the shape of the data in the response payload. May be an empty list for a scalar result.x-activestorage-count: The number of non-missing array elements operated on while performing the requested reduction. This header is useful, for example, to calculate the mean over multiple requests where the number of items operated on may differ between chunks.On error, an HTTP 4XX (client) or 5XX (server) response code will be returned, with the response body being a JSON object of the following format:
+{
+ "error": {
+ // Main error message
+ "message": "error receiving object from S3 storage",
+
+ // Optional list of lower-level errors, with the root cause last
+ "caused_by": [
+ "IO error",
+ "unexpected end of file"
+ ]
+ }
+}
+The scripts/client.py provides an example Python client and Command Line Interface (CLI).
+ + + + + + +Reductionist is written in Rust, a language that is rapidly gaining popularity for a variety of use cases. +It provides high level abstractions with low runtime overhead, a modern toolchain, and has a unique approach that provides safe automatic memory management without garbage collection. +While the Rust standard library is not as comprehensive as some other "batteries included" languages, the crates.io ecosystem is relatively mature and provides a number of de-facto standard libraries. +Reductionist is built on top of a number of popular open source components.
+A few properties make it relatively easy to build a conceptual mental model of how Reductionist works.
+The more challenging aspects of the system are the lower level details of asynchronous programming, memory management, the Rust type system and working with multi-dimensional arrays.
+A diagram of the request processing pipeline is shown in Figure 1.
+
+
The "Perform numerical operation" step depends on the type of numerical operation being performed. +A diagram of this step for the sum operation is shown in Figure 2.
+
+
Axum is an asynchronous web framework that performs well in various benchmarks and is built on top of various popular components, including the hyper HTTP library.
+It integrates well with Tokio, the most popular asynchronous Rust runtime, and allows us to easily define an API route for each operation.
+Extractors make it easy to consume data from the request in a type-safe way.
+The operation request handler is the operation_handler function in src/app.rs.
The JSON request data is deserialised into the RequestData struct defined in src/models.rs using the serde library.
+Serde handles conversion errors at the type level, while further validation of request data invariants is performed using the validator crate.
Object data is downloaded from the object store using the AWS SDK.
+The S3Client struct in src/s3_client.rs provides a simplified wrapper around the AWS SDK.
+Typically we will be operating on a "storage chunk", a hyperslab within the larger dataset that the object contains.
+In this case a byte range is specified in the S3 GetObject request to avoid downloading the whole object.
+The AWS SDK is asynchronous and does provide a streaming response, however we read the whole storage chunk into memory to simplify later stages of the pipeline.
+Storage chunks are expected to be small enough (O(MiB)) that this should not be a problem.
Construction of aws_sdk_s3::Client structs is a relatively slow task.
+A key performance improvement involves the use of a shared client object for each combination of object store URL and credentials.
+This is implemented using the S3ClientMap in src/s3_client.rs and benchmarked in benches/s3_client.rs.
Downloaded storage chunk data is returned to the request handler as a Bytes object, which is a wrapper around a u8 (byte) array.
When a variable in a netCDF, HDF5 or Zarr dataset is created, it may be compressed to reduce storage requirements.
+Additionally, prior to compression one or more filters may be applied to the data with the aim of increasing the compression ratio.
+When consuming such data, Reductionist needs to reverse any compression and filters applied.
+The filter pipeline is implemented in src/filter_pipeline.rs.
First, if a compression algorithm is specified in the request data, the storage chunk is decompressed using the same algorithm.
+Currently the Gzip and Zlib algorithms are supported using the flate2 and zune-inflate libraries respectively.
+This mix of libraries was chosen based on performance benchmarks in benches/compression.rs.
+Compression is implemented in src/compression.rs.
Next, if any filters are specified in the request data, they are decoded in reverse order.
+Currently the byte shuffle filter is supported.
+This filter reorders the data to place the Nth bytes of each data value together, with the aim of grouping leading zeroes.
+The shuffle filter is implemented in src/filters/shuffle.rs, and has several optimisations including loop unrolling that were benchmarked using benches/shuffle.rs.
Here the implementation becomes specific to the requested operation (min, max, etc.).
+This is achieved using the Operation trait defined in src/operation.rs.
/// Trait for active storage operations.
+///
+/// This forms the contract between the API layer and operations.
+pub trait Operation {
+ /// Execute the operation.
+ ///
+ /// Returns a [models::Response](crate::models::Response) object with response data.
+ ///
+ /// # Arguments
+ ///
+ /// * `request_data`: RequestData object for the request
+ /// * `data`: [`Vec<u8>`] containing data to operate on.
+ fn execute(
+ request_data: &models::RequestData,
+ data: Vec<u8>,
+ ) -> Result<models::Response, ActiveStorageError>;
+}
+This interface accepts the request data and a byte array containing the storage chunk data in its original byte order.
+On success, it returns a Response struct which contains a byte array of the response data as well as the data type, shape and a count of non-missing elements in the array.
A second NumOperation trait with an execute_t method handles the dynamic dispatch between the runtime data type in the request data and the generic implementation for that type.
Each operation is implemented by a struct that implements the NumOperation trait.
+For example, the sum operation is implemented by the Sum struct in src/operations.rs.
+The Sum struct's execute_t method does the following:
ndarray::ArrayView onto the original array viewsum and len methods to take the sum and element countThe procedure for other operations varies slightly but generally follows the same pattern.
+The ActiveStorageError enum in src/error.rs describes the various errors that may be returned by the Reductionist API, as well as how to format them for the JSON error response body.
+Low-level errors are converted to higher-level errors and ultimately wrapped by ActiveStorageError.
+This is a common pattern in Rust and allows us to describe all of the errors that a function or application may return.
Reductionist configuration is implemented in src/cli.rs using the clap library, and accepts command line arguments and environment variables.
Reductionist supports optional restriction of resource usage.
+This is implemented in src/resource_manager.rs using Tokio Semaphores.
+This allows Reductionist to limit the quantity of various resources used at any time:
There is particular friction between the asynchronous and synchronous types of work in the system. +Axum and Tokio very efficiently handle the asynchronous aspects such as the HTTP server and S3 object download. +The other work such as decompression, filtering and numerical operations are more CPU-bound, and can easily block the Tokio runtime from efficiently handling asynchronous tasks. +Two alternative methods were developed to alleviate this issue.
+Limited benchmarking was done to compare the two approaches, however the first appeared to have lower overhead. +The second approach may leave the server more responsive if more CPU-heavy operations are used in future.
+Prometheus metrics are implemented in src/metrics.rs and are exposed by the Reductionist API under the /metrics path.
+These include:
Reductionist integrates with Jaeger, a distributed tracing platform. +Various sections of the request processing pipeline are instrumented with spans, making it easy to visualise the relative durations in the Jaeger UI. +Testing with a sum over some CMIP6 temperature data, this showed that in terms of wall clock time, the S3 storage chunk download takes the majority of the time, followed by decompression, byte shuffle, and finally the actual numerical operation.
+Flame graphs created using flamegraph-rs were useful to visualise which parts of the code consume the most CPU cycles. +This was useful to determine where to focus performance improvements, and showed that decompression is the most CPU-heavy task.
+ + + + + + +\n {translation(\"search.result.term.missing\")}: {...missing}\n
\n }\n \n