CFS multi-core/SMP Strategies

[jwberck]

Hey all,

I'm pasting a question that I asked @skliper at the FSW workshop related to multi-core/SMP strategies attached below. I'm hoping we can get a discussion going on various experiences that folks have had here, suggestions for those going into the future for utilizing multi-core/SMP, and potential feature suggestions for CFS.

My Question
"From my limited perspective, and feel free to correct me, it seems like cFS was designed in a way that is biased towards comprehending single core systems. There have been additions to help with multicore, but I get the sense that there is really some foundational architecture in place that is geared towards single. I can see some places that would help multi-core performance, like reducing some of the lock scope of various core apps, but I'm curious what sort of guidance you'd give going forward into a multi-core world. We haven't hit too many issues yet as we don't have much sequential coupling between apps and there is no explicitly parallel processes going on. But I am expecting some hidden traps in dark corners going forward as we attempt parallel processing around various computational bottlenecks in a soft real time system. Any advice going forward with cFS into the parallel multi-core world?"

@skliper 's response
"There's been a lot of interest lately. You're definitely correct that cFS originated on single core systems but we are starting to have an increasing number of stakeholders using SMP. There's a large range of SMP concepts floating about from very simple static core assignment to power and failure management, dynamic balancing, parallel processing, voting, etc. For the simplest case there are now hooks in OSAL to register a function call for task startup that could be used to assign to cores, the simplest concept is just name based (call your app XXX_C1 to lock to core 1 or similar) using the same approach as how task names are assigned on Linux. I encourage submitting issues for any cases where we could reduce lock scope, we are definitely interested in reducing contention. We have also had users run into decreased message queue performance across cores on at least one embedded system w/ an RTOS, so there's some interest in a custom queue implementation (IIRC there's an issue open in the cFE and/or OSAL repo). I do often encourage projects to keep the design as simple as possible, assign tasks to cores in sensible groups, and pay attention to everything that crosses the cores or is shared (keep anything that could run into contention out of any time critical operations). I get nervous whenever a project just lets things float or anticipates the numbers of cores directly multiplies performance."

Could anyone else share their experiences here?

[jphickey]

There's a lot in that question, I think it needs to be broken apart somewhat:

• What are your goals of using a multi-core design? Are you just looking to get more parallelism in your tasks/threads (the most typical goal) or are you looking for something more complex?
• Are you referring to the CFE framework itself, or the CFS application suite?

Note with respect to apps, that each app is independently developed, and nearly all of which were initially designed when single-core processors were the norm. Many have been updated since to do proper locking, but depending on what app you are using, you may or may not find its design well suited for use on SMP.

Other comments:

• The default behavior on most SMP kernels is to allow tasks to float between cores - that is, whenever a task is ready to run, it is assigned to an available core to run it, and may freely switch between them each time it gets rescheduled. This generally gets you the most parallelism possible because it keeps all available cores as busy as possible, and should "just work" as long as the shared objects are properly synchronized w/mutex/semaphore or similar.
• Some users don't like floating because switching cores may add latency and disrupts cache, causing non-deterministic timing for time sensitive tasks. The solution typically is to pin tasks to a specific core or cores (perhaps the set of cores that share an L2 cache) - but this is all very platform specific. There are hooks/callbacks from OSAL that one can add to the PSP that allow tasks to be pinned, if need be. None of this will be part of a default configuration though, because it is very platform-dependent, and use-case dependent.
• OSAL does not currently expose anything like a spinlock for protecting the small shared object accesses - so a normal mutex is used everywhere instead. This may conceivably have a negative performance impact for some circumstances (task switch may occur where it was unexpected) but as of yet this hasn't risen to being a point of concern big enough for a stakeholder to warrant implementing this API. It's been on the to-do list, but there needs to be a real driver for it to make it happen.

I'll reiterate the previous comment - please submit issues for any specific problems or concerns you encounter when running CFS on SMP platforms. The intent is that SMP should work, but again it depends on what you are trying to accomplish as well.

[skliper]

Note I did just submit an example for setting thread cpu affinity at startup from the PSP based on task name matching. See nasa/PSP#341