FPP, array size limitation for large byte buffer (pixel buffer)

[ThibFrgsGmz]

Hello,

I would like to specify in my topology that two components must exchange an image (pixel buffer) containing a large number of pixels (let's take for example 16 million U32).

    constant ImageWidth = 4000
    constant ImageHeight = 4000
    array ImagePixelBuffer = [ImageWidth * ImageHeight] U32

I create a port to pass an image from an acquisition component to a processing component:

    port ImageProcessor(
        image_pixel_buffer : ImagePixelBuffer @< Acquired image to process
    )

Unfortunately, I run into a limitation in FPP that does not allow arrays larger than 256: An array value can have a maximum of 256 elements.

How to solve this problem? Define the type in C++ and reference it in FPP?

[timcanham]

We typically pass around images using the Fw::Buffer pattern. You create a pool of image buffers in Svc/BufferManager, and then use the Fw/Buffer port to send them around and return them to Svc/BufferManager when you are done.

[ThibFrgsGmz]

Thanks a lot @timcanham for your quick answer.
I will look for a concrete example in the code base. I'll come back to you if I have any doubts about the understanding/implementation!

[ThibFrgsGmz]

@timcanham What would be the key things to consider when making a trade-off between an array and a Fw.Buffer ?
The advantage of the Fw.Buffer would be mainly the possibility to have a rotating buffer managed by the BufferManager?

[timcanham]

The key thing IMHO is the size. If it is a relatively small array and it makes sense to pass it by value through message queues, that gives you clear ownership of the data. If it is a larger data set like an image, it makes more sense to pass around Fw::Buffer which is light-weight to copy and manage ownership.

[ThibFrgsGmz]

Ok perfect thanks!

[LeStarch]

As a note, we followed @timcanham pattern of passing buffers for our imaging in the upcoming system reference.

https://github.com/fprime-community/fprime-system-reference/tree/pelmini-branch/SystemReference/Payload

The library we use to capture and store images is OpenCV, so there is a little uncontrolled dynamic memory. We will fix that, but our images are passed between components using Fw::Buffers allocated with the Svc::BufferManager and that memory is controlled.

[ThibFrgsGmz]

Looks awesome but the link does not exist :(
Is this repo private ?

[LeStarch]

You are right. It is private. I will add you for now. We intend to have that open and viewable by the end of this month.

[ThibFrgsGmz]

Ok, thanks a lot, I'm browsing a bit and it looks like a great job!
I can't wait for the community to have access and play with it 😀

[ThibFrgsGmz]

@pelmini @LeStarch Why did you reuse the BufferManager used for the FileUplink to manage the images in the SystemReference repo?
The semantics of the component means to me that the images are transmitted by the GDS but you use it for any data transmitted to the embedded systems (sensors, GDS,...)?

@timcanham As I understand it, there are two types of buffer management, one dynamic with BufferManager and one static with StaticMemory.

With BufferManager, we have the ability to specify multiple heterogeneous memory locations to hold FIFOs of different data structures and sizes.
The flexibility of this component is done via a malloc (customizable implementation depending on the HW platform) and the Svc::BufferManager::BufferBins object to size the different memory areas.

As for StaticMemory, I feel that in its current state it has severe limitations and seems unusable in the context of my application.

This component would simply be a global pool of static memory of a size configurable by global deployment-specific parameters (StaticMemoryAllocations * STATIC_MEMORY_ALLOCATION_SIZE bytes, see nasa/fprime/Svc/StaticMemory/docs/sdd.md).

In my application case, I would have 3 cameras with different image sizes (512x512, 1024x1024, 2048x2048) and I want to statically allocate memory areas to store and process the images.

The memory areas for these images are not coalescing because I need to map them to different memory addresses.

Using the StaticMemory component seems compromised to me:

• I have to allocate more size than I need: 3 (cameras) * 2048*2048 (maximum size of an image).

• The StaticMemory pool is coalescing, I would not be able to map the pool bins to specific addresses.

• there would be the possibility to instantiate only one StaticMemory component per deployment due to the nature of its configuration parameters.

Did I correctly understand the implications of these components?

[SterlingPeet]

I think the StaticMemory component is better represented as an example blueprint than a flexible and full featured component. Is there a reason why you would not be interested in creating a new component that is derived from StaticMemory, but where the static memory size is passed in as an initialization parameter? It will require a dynamic allocation of that memory at runtime, once, before starting the main application rather than statically defining it at compile time.

The alternative option would be to create three derived components that all have their own memory size definition at compile time, but that seems like a blatant violation of the DRY principal to me.

[LeStarch]

Yes, @SterlingPeet is right. The StaticMemory component was created to have a very basic memory management scheme that followed the same interface as the BufferManager. At the time of theStaticMemory component's creation, the BufferManager had some sever limitations that prevented it from being used in uplink and downlink. However, we needed something that could allow uplink and downlink to use the correct memory interface as we refactored those before revising the BufferManager. StaticMemory has limitations in configuration (as you noted) but also has limitations in usage (e.g. only one allocation at a time).

The current "revised" BufferManager is free of the limitations of the original design and of the limitations of StaticMemory. It replace StaticMemory as the memory manager for uplink and downlink in the final SystemReference design as it is more suitable for the task since it can allow multiple allocations at a time, we can therefore have asynchronous components in uplink and downlink (e.g. an async radio).

Why did we use the fileUplinkBufferManager for the imaging system? Ease of prototyping. It was easier to get up and running with an existing buffer manager than it was to instantiate a new one. @timcanham recommends one buffer manager per subsystem (e.g. one for uplink and downlink, one for payload image acquisition, etc) and in the final system reference we will have this sorted out.

The trade-offs are:

1. More buffer managers means more memory "sitting around" when not in use (less reuse)
2. Less buffer managers means more contention over the existing memory (more competition).

Since the sizes of the images are drastically larger than that needed for uplink and downlink @timcanham's suggestion is on-point and we will likely split off payload memory management from uplink and downlink buffer management in the final released version.

[LeStarch]

For your described use case (if I followed correctly) of statically allocated memory at fixed addresses, you could adapt the StaticMemory component to take in a Fw::MemAllocator to setup its memory regions. You would then build a memory allocator that returns the fixed addresses back on the allocate call. This is similar to what @SterlingPeet described using a mem allocator at initialization to provide the buffers, but adds in a custom mem allocator to provide your custom address allocations.

[ThibFrgsGmz]

OK, thank you so much @LeStarch for this information, I'll see what I can do!