Example firmware for running SWIFT-YOLO models on Grove Vision AI Module V2

[ThomasBroch]

Describe the feature
Please provide example firmware code for running swift yolo models, made with model assistant, to run them on the Grove Vision Module V2.

Motivation
Through model assistant, swift yolo models can be trained that can be deployed to the Grove Vision AI Module V2. There appears to be no firmware source code example available for running these swift yolo models.

There are various firmware example apps available in the firmware repository, but there don't appear to be any for running swift yolo. Whenever you load a swift yolo through the sensecraft web toolkit, it also downloads and flashes a firmware image to run the model, but the source code of that firmware is not available.

Additional context
We are working on a custom implementation where we would like to send an image over serial to the grove, rather than using a camera module. For that, we need to write our own firmware, but we have not been able to find example firmware for running swift yolo.

I have been in contact about this with support and product line managers. They asked me to detail my issue here.

[iChizer0]

Hi @ThomasBroch, we have written a minimal sample code for you, hopefully it resolves your confusion.

static void __task(void*) {
    auto device = Device::getInstance();

    auto sensors = device->getSensors();
    MA_ASSERT(!sensors.empty());

    auto sensor = sensors.front();
    auto rc     = sensor->init(1);
    MA_ASSERT(rc == MA_OK);

    auto models = device->getModels();
    MA_ASSERT(!models.empty());

    auto model = models.front();
    MA_ASSERT(model.addr != nullptr);

    auto engine = ma::engine::EngineTFLite();
    rc          = engine.init();
    MA_ASSERT(rc == MA_OK);

    rc = engine.load(model.addr, model.size);
    MA_ASSERT(rc == MA_OK);

    auto algo = ma::ModelFactory::create(&engine);
    MA_ASSERT(algo != nullptr);

    auto camera = static_cast<ma::Camera*>(sensor);
    rc = camera->startStream(ma::Camera::StreamMode::kRefreshOnReturn);
    MA_ASSERT(rc == MA_OK);
    
    printf("Running algo: %d\n", algo->getType());
    for (;;) {
        auto frame = ma_img_t{};

        rc = camera->retrieveFrame(frame, MA_PIXEL_FORMAT_AUTO);
        if (rc != MA_OK) {
            continue;
        }

        switch (algo->getType()) {
            case MA_MODEL_TYPE_PFLD:
            case MA_MODEL_TYPE_IMCLS:
            case MA_MODEL_TYPE_YOLOV8_POSE:
                // ...
                break;

            case MA_MODEL_TYPE_FOMO:
            case MA_MODEL_TYPE_YOLOV5:
            case MA_MODEL_TYPE_YOLOV8:
            case MA_MODEL_TYPE_NVIDIA_DET:
            case MA_MODEL_TYPE_YOLO_WORLD: {
                auto D = static_cast<ma::model::Detector*>(algo);

                rc = D->run(&frame);
                if (rc != MA_OK) {
                    continue;
                }

                auto results = D->getResults();
                for (auto& result : results) {
                    result.x *= frame.width;
                    result.y *= frame.height;
                    result.w *= frame.width;
                    result.h *= frame.height;
                    result.score *= 100;
                }

                printf("Results size: %d\n", static_cast<int>(
                    std::distance(results.begin(), results.end())
                ));
                for (auto& result : results) {
                    printf("x: %f, y: %f, w: %f, h: %f, score: %f\n", result.x, result.y, result.w, result.h, result.score);
                }
            }

            default:
                // ...
                break;
        }

        camera->returnFrame(frame);
    }

    camera->stopStream();
}

You can test this by replacing the corresponding function in the following branch with the code above:

https://github.com/Seeed-Studio/sscma-example-we2/blob/7413c43ad5de0883e7117495e47b7e848640034f/EPII_CM55M_APP_S/app/scenario_app/sscma/sscma.cpp#L13

If you wish to customize the data source of your images, we recommend you to read the source code of https://github.com/Seeed-Studio/SSCMA-Micro/tree/refactor/sscma-deploy. And if there's any further questions, please contact us again, thank you!

[ThomasBroch]

@iChizer0 Thanks. I was not aware of the Seeed fork of the firmware repository. Earlier, I was only pointed to the Himax parent repository, which does not include the sscma example app.

With the sscma example app, the repository does not appear to include the actual SSCMA micro SDK, even after updating the submodules:

app/scenario_app/sscma/sscma.cpp:1:10: fatal error: sscma.h: No such file or directory
    1 | #include <sscma.h>

The library directory only contains sscma_micro_porting. Do you have any instructions for including the actual SSCMA micro SDK? I have not been able to find any documentation about this.

(If you want I can also start a new issue for this on the firmware repository.)

[iChizer0]

It looks like there are some problems with Git's submodule, and I've just committed a fix to the main branch.

Seeed-Studio/sscma-example-we2@b8f11d5

You can try switching to the latest main branch and fetch the submodules again, thanks.

git submodule update --recursive --init

[ThomasBroch]

Thnx @iChizer0. There also appears to be an issue with the cJSON fetching mechanism. The fetch_cjson.sh script does not get called during the make process. So I called the script manually, supplying the component hash from CMakelist.txt. That did fetch cJSON. However, the make process still failed, being unable to locate cJSON.

./library/sscma_micro/sscma/server/at/codec/ma_codec_json.h:4:10: fatal error: cJSON.h: No such file or directory
    4 | #include <cJSON.h>
      |          ^~~~~~~~~
compilation terminated.

[iChizer0]

I'm sorry, but this commit solves the problem of missing dependencies, please try again, thanks.

Seeed-Studio/sscma-example-we2@461496f

[ThomasBroch]

@iChizer0 Thnx! It builds now. Do you have any documentation available about the various methods like getModels. The SDK documentation does not appear to list it.

Your fast help so far is greatly appreciated!

[iChizer0]

Yeah, we have auto-generated documentation which has now been updated and you can view it here: ma::Device Class Reference.