Onion Omega2 and musl support

[aditsachde]

Hi! I'm working on a project that uses the Onion Omega2 linux module. We have set up a custom toolchain using the template.

There is a two line patch required, due to the the toolchain using musl instead of glibc. The main incompatibility is the pthread_attr_setaffinity_np function, which is not supported. We chose to just disable it, as we do not need it.

What is the best way to manage these changes without a fork of fprime? What would the process for upstreaming musl support into fprime look like?

[Joshua-Anderson]

I've never seen anybody using musl in an embedded context before but I guess it'll be a library size improvement over glibc, so why not!

What would the process for upstreaming musl support into fprime look like?

Let's get these matches merged in! The include patch in IpSocket.cpp I think could be merged in as is.

As for avoiding pthread_attr_setaffinity_np, we should consider switching over to using pthread_setaffinity_np, which is supported by musl.

If we want to prevent musl specific regressions we also probably want to add a CI job that compiles fprime and runs the unit tests using musl (maybe with an alpine image?).

[Joshua-Anderson]

The function where it is called has the following parameter: pthread_attr_t& att, whereas pthread_setaffinity_np needs pthread_t. I'm not sure if swapping is easily doable or would require a breaking change.

It's not a public interface, just an internal function for setting affinity, so we can modify it without breaking anything.
However, I realized that we were setting CPU affinity before starting the thread, which wouldn't be possible with pthread_setaffinity_np. I'm wondering if there's another musl supported way to set task cpu affinity before launching the thread.

[aditsachde]

I've taken an attempt making this work by disabling cpu affinity support when using musl, but it requires some way to know musl is currently being used. There are two options I see:

1. Detect musl or glibc usage.
2. Add a compile time definition to the toolchain cmake.

Fprime suggests to construct a platform file if the user wants to set compile flags. However, the libc being used is defined by the toolchain, not the platform, so it seems that this flag would make more sense being set within the toolchain file. Detecting musl usage is possible, but seems to be more brittle than a compile flag.

What is the preferred solution?

[Joshua-Anderson]

Does musl really have no thread cpu affinity support? If it does have thread cpu affinity support, we should try and use it.

[aditsachde]

Musl tries to conform to posix specifications as much as possible. pthread_attr_setaffinity_np is not defined in posix, and so musl unfortunately does not implement it. The option musl provides for cpu affinity is pthread_setaffinity_np, but as you pointed out, fprime sets the cpu affinity prior to starting the thread, which would not be possible using this function.

It doesn't seem too difficult to switch the order and set cpu affinity after creating the thread. It would however bring up the question of what to do if the thread is successfully created, but setting the cpu affinity fails.

[Joshua-Anderson]

@LeStarch What are your thoughts here?

Is it better to set thread affinity before task creation and be non-portable but more deterministic or set thread affinity shortly after creation and be portable?

[timcanham]

You can also treat this like a non-Linux system and write custom Os implementations, similar to VxWorks. Then you can tailor the calls for musl.