-
Notifications
You must be signed in to change notification settings - Fork 6.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
drivers: video: video_stm32_dcmi: Use video buffers for DCMI buffer. #84446
base: main
Are you sure you want to change the base?
Conversation
aa2a9d8
to
956ea7f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(context for other readers =>) It seems like the "continuous mode" of DCMI uses a different model where processing happens in the leap between two frames, rather than use a different buffer every time. That could be improved in the future maybe, but until then, allocating a buffer locally and memcpy()
-ing it on every vbuf->data
coming from the API seems like the way to go.
See inline comment for a proposal for how to integrate it... Thank you!
drivers/video/video_stm32_dcmi.c
Outdated
#if defined(CONFIG_VIDEO_BUFFER_USE_SHARED_MULTI_HEAP) | ||
#include <zephyr/multi_heap/shared_multi_heap.h> | ||
#define VIDEO_STM32_HEAP_ALLOC(align, size, timeout) \ | ||
shared_multi_heap_aligned_alloc(CONFIG_VIDEO_BUFFER_SMH_ATTRIBUTE, align, size) | ||
#define VIDEO_STM32_HEAP_FREE(block) shared_multi_heap_free(block) | ||
#else | ||
K_HEAP_DEFINE(video_stm32_buffer_pool, CONFIG_VIDEO_BUFFER_POOL_SZ_MAX); | ||
#define VIDEO_STM32_HEAP_ALLOC(align, size, timeout) \ | ||
k_heap_aligned_alloc(&video_stm32_buffer_pool, align, size, timeout); | ||
#define VIDEO_STM32_HEAP_FREE(block) k_heap_free(&video_stm32_buffer_pool, block) | ||
#endif |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like this snippet is the same as here save for the symbol name:
zephyr/drivers/video/video_common.c
Lines 16 to 24 in e4389a2
#define VIDEO_COMMON_HEAP_ALLOC(align, size, timeout) \ | |
shared_multi_heap_aligned_alloc(CONFIG_VIDEO_BUFFER_SMH_ATTRIBUTE, align, size) | |
#define VIDEO_COMMON_FREE(block) shared_multi_heap_free(block) | |
#else | |
K_HEAP_DEFINE(video_buffer_pool, CONFIG_VIDEO_BUFFER_POOL_SZ_MAX*CONFIG_VIDEO_BUFFER_POOL_NUM_MAX); | |
#define VIDEO_COMMON_HEAP_ALLOC(align, size, timeout) \ | |
k_heap_aligned_alloc(&video_buffer_pool, align, size, timeout); | |
#define VIDEO_COMMON_FREE(block) k_heap_free(&video_buffer_pool, block) | |
#endif |
How about moving the video_common.c
macros (except K_HEAP_DEFINE()
) to video_common.h
, and adding a buffer_pool
argument to specify which buffer pool is targeted?
That way, both video_common.c
and video_some_driver.c
can use the same implementation.
#if defined(CONFIG_VIDEO_BUFFER_USE_SHARED_MULTI_HEAP) | |
#include <zephyr/multi_heap/shared_multi_heap.h> | |
#define VIDEO_STM32_HEAP_ALLOC(align, size, timeout) \ | |
shared_multi_heap_aligned_alloc(CONFIG_VIDEO_BUFFER_SMH_ATTRIBUTE, align, size) | |
#define VIDEO_STM32_HEAP_FREE(block) shared_multi_heap_free(block) | |
#else | |
K_HEAP_DEFINE(video_stm32_buffer_pool, CONFIG_VIDEO_BUFFER_POOL_SZ_MAX); | |
#define VIDEO_STM32_HEAP_ALLOC(align, size, timeout) \ | |
k_heap_aligned_alloc(&video_stm32_buffer_pool, align, size, timeout); | |
#define VIDEO_STM32_HEAP_FREE(block) k_heap_free(&video_stm32_buffer_pool, block) | |
#endif | |
#if defined(CONFIG_VIDEO_BUFFER_USE_SHARED_MULTI_HEAP) | |
#include <zephyr/multi_heap/shared_multi_heap.h> | |
#define VIDEO_STM32_HEAP_ALLOC(buffer_pool, align, size, timeout) \ | |
shared_multi_heap_aligned_alloc(CONFIG_VIDEO_BUFFER_SMH_ATTRIBUTE, align, size) | |
#define VIDEO_STM32_HEAP_FREE(buffer_pool, block) shared_multi_heap_free(block) | |
#else | |
K_HEAP_DEFINE(video_stm32_buffer_pool, CONFIG_VIDEO_BUFFER_POOL_SZ_MAX); | |
#define VIDEO_STM32_HEAP_ALLOC(buffer_pool, align, size, timeout) \ | |
k_heap_aligned_alloc(buffer_pool, align, size, timeout); | |
#define VIDEO_STM32_HEAP_FREE(buffer_pool, block) k_heap_free(buffer_pool, block) | |
#endif |
I am asking for your feedback on this, and if this does not make sense, the proposed solution also makes sense to me!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about moving the video_common.c macros (except K_HEAP_DEFINE()) to video_common.h,
Either way is fine with me. There's no video_common.h
though, new file? Or do you mean video.h
?
Also we'll need something like this in drivers if we add the pool arg:
#if defined(CONFIG_VIDEO_BUFFER_USE_SHARED_MULTI_HEAP)
#define VIDEO_BUFFER_POOL (0)
#else
K_HEAP_DEFINE(VIDEO_BUFFER_POOL, CONFIG_VIDEO_BUFFER_POOL_SZ_MAX*CONFIG_VIDEO_BUFFER_POOL_NUM_MAX);
#endif
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's no video_common.h though, new file? Or do you mean video.h?
Ah you are right. This would be a new file, or if you believe you could ever want to call that from the application, this can justify to put it in the public API in video.h
, in which case, it becomes part of the public video API rather than an implementation detail (needs deprecation warning if API is changed, ideally unit tests too, etc).
Also we'll need something like this in drivers if we add the pool arg:
That is a good idea! That would allow to skip the pool
arg altogether if there is a local macro that defines the VIDEO_BUFFER_POOL
to use locally. More convenient that way!
An example of "xxx_common.h" file for USB:
https://github.com/zephyrproject-rtos/zephyr/blob/main/drivers/usb/udc/udc_common.h
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. How about now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for taking the effort of refactoring this in generic APIs!
8dec094
to
2452a5f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is possible that other collaborators will have a better outlook of how to handle this.
At my level, it looks like the best way to implement it without completely reorganizing the video APIs (i.e. switch to net_buf
or rtio
for buffer management).
Thank you very much!
2452a5f
to
535c1e0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, I missed it.
Just a first look, perhaps I missed some thing but I don't understand why you need to expose the |
Because as it stands, One way to fix this is to require a minimum of two buffers for this driver, but I don't know how to do that, or if it's even possible, is it? |
Ok, I understand the context now.
So, I think you just need to do it in the DCMI driver. Refactoring the video_common to expose the macro VIDEO_HEAP_ALLOC is not the right way to do here because
Even if we want a "common" macro to allocate buffers on any heap, this should be defined system-wide, not inside the video subsystem (but again, it does not save much code lines) |
You mean duplicate the macro? The initial PR did that exactly, but then I was asked to move it to common header. Are we all in agreement this time that it needs to be duplicated?
It does, because without it, we'll have to duplicate the macro more or less. Alternatively, requiring a minimum number of buffers (in the case of stm32 >= 2) for stm32 is another valid option, and then we can use |
Sorry for that, cc @josuah (?). It is used only in one place so you maybe don't need to define the macro, just
? but if it needs to be duplicated, we do it. There are code duplication existing in Zephyr, especially when you look at the .conf, overlay. As explained above, There are examples such as a display driver needs a specific display heap for buffer allocation as here so it does not mean that display driver should use the refactored VIDEO_HEAP_ALLOC macro (?).
In general, it could be an option. The DCMI can use the common video buffer pool for its personal purpose as well. The A similar option which was discussed here is to use the system heap instead of the driver heap and extend it in the driver with CONFIG_HEAP_MEM_POOL_ADD_SIZE_XXX but if I remember well it does not add the additiobnal required size but specifies the whole size ?? (the discussion was a long time ago, I need to recall it). |
Actually it should, because it reserves a chunk of memory that it may not use at all.
I thought about this and have a better idea. What if |
Instead of reserving a static (possibly unaligned) buffer for DCMI, this patch reserves and holds one of the video buffers to use as the main DCMI buffer. This buffer will be aligned (using the alignment specified in the config) and will either be allocated from `video_common` pool or a shared multi-heap (if enabled). Signed-off-by: Ibrahim Abdalkader <[email protected]>
I think you misunderstood my point here. I mean the display driver could use a common macro like HEAP_ALLOC() defined system wide (to save some line of codes) but it should not include video_common.h and use VIDEO_HEAP_ALLOC() because VIDEO_HEAP_ALLOC() is for video subsystem. To be clearer, take a look at the signature of the newly refactored VIDEO_HEAP_ALLOC:
there is nothing specific to video anymore, so that's why it should be |
535c1e0
to
ae15878
Compare
I see. Well I just need to solve my immediate problem right now: Use SMH if enabled. I imagine adding a general purpose macro/public API will require reviews docs, tests etc.. Please see the updated commit, if it's not good for any reason I'll duplicate the macro in |
Thanks. I need some time to understand the context / purpose of the additional buffer in the DCMI driver (I missed the review of this driver). Otherwise, @CharlesDias may have a better look. |
No worries. FWIW, there's no reason not to use the FIFO to pass buffers between driver application without |
Thanks all for this in-depth study!
It is possible to declare it, although, not taken advantage of through the samples or API: zephyr/include/zephyr/drivers/video.h Line 98 in 9e08560
That sounds good, it would be a more lightweight intermediate solution for a more general fix coming later.
That is how most video drivers work it seems, and if it is possible to make DCMI work that way, that avoids the problem of local video memory allocation in the driver.
The way the buffers are currently loaded/unloaded: zephyr/drivers/video/video_stm32_dcmi.c Lines 68 to 83 in 35abb31
I do not see any API call that allows to update that
Maybe the hardware does not allow to update the buffer without stopping/starting the engine? |
I've already implemented this and updated the PR, no local video buffer allocation but still uses
The DMA does allow updating the target address when not in use, in general or just in double buffer mode can't remember, but either way this is a different issue, which I'm not trying to fix. However, this PR is a step in the right direction: you need to use video buffers before you can remove the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I missed the update. Very good first step indeed!
I prefer it over what I originally suggested (previous LGTM) as I did not think of this idea.
+1 for my part!
data->vbuf = k_fifo_get(&data->fifo_in, K_NO_WAIT); | ||
|
||
if (data->vbuf == NULL) { | ||
LOG_ERR("Failed to dequeue a DCMI buffer."); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After this commit, if a video buffer is not passed before start(), this would return an error.
I think that all examples in main
would still work:
here: 2 buffers
https://github.com/zephyrproject-rtos/zephyr/blob/main/samples/drivers/video/tcpserversink/src/main.c#L41
https://github.com/zephyrproject-rtos/zephyr/blob/main/samples/drivers/video/capture_to_lvgl/src/main.c#L22
or otherwise, CONFIG_VIDEO_BUFFER_POOL_NUM_MAX
https://github.com/zephyrproject-rtos/zephyr/blob/main/samples/drivers/video/capture/src/main.c#L85
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the applications should still work, mine still works without changes:
App: allocates 2 buffers.
App: enqueues 2 buffers in fifo_in.
App: starts streaming.
Driver: dequeues buffer 1 from fifo_in and holds it for use as DCMI buffer.
Driver: Frame ready, dequeue buffer 2 from fifo_in, memcpy, enqueue in fifo_out.
Hi, @iabdalkader. Thank you for your contribution! This is a great improvement! :) @ngphibang and @josuah, is there another way to release the video buffers besides using the Additionally, I tested the capture_to_lvgl sample on MiniSTM32H743 and still worked. |
Thank you for testing it!
If the application decides to use a AFAICT, the current APIs give the driver the freedom of how many buffers it holds and in which order they are released. zephyr/drivers/video/video_common.c Line 71 in e4389a2
The video buffer variables are not exported, there is no way to access it from the outside other than the provided functions. |
Instead of reserving a static (possibly unaligned) buffer for DCMI, this patch reserves and holds one of the video buffers to use as the main DCMI buffer. This buffer will be aligned (using the alignment specified in the config) and will either be allocated from
video_common
pool or a shared multi-heap (if enabled).