Step 4: Develop, test and verify output of argmax_print custom plugin

../../../_images/resnet50_application_simaaisrc_argmax_print.jpg

As a final step, we need to perform argmax to select the highest probability. The output from the network is an array of 1000 probabilities. Thus, we should expect an output of 1000 fp32 values, or 4000 bytes total (fp32 => 4 bytes).

To do so, we will create a custom plugin where we will:

  1. Take in the buffer from the simaaiprocesscvu (running the SIMA_DETESS_DEQUANT CVU graph) plugin.

  2. Perform the argmax operation on it.

  3. Print the output class index to a console.

Creating a custom Plugin

On your host machine’s Palette docker install, go to SiMa’s pluginzoo directory, copy the simple_overlay plugin example and compile as a test:

sima-user@docker-image-id:~$ source /opt/poky/4.0.10/environment-setup-cortexa65-poky-linux
sima-user@docker-image-id:~$ cd /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst$ cp -r examples/opencv/simple_overlay ./argmax_print
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst$ cd argmax_print/
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ mkdir build && cd build
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ cmake ..
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ make
    [ 50%] Building CXX object CMakeFiles/argmax_print.dir/payload.cpp.o
    /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/payload.cpp: In member function ‘void UserContext::run(std::vector<Input>&, std::span<unsigned char>)’:
    /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/payload.cpp:57:71: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
    57 |     cv::imencode(".jpg", image, reinterpret_cast<std::vector<uchar>&>(output));
        |                                                                       ^~~~~~
    [100%] Linking CXX shared library libargmax_print.so
    [100%] Built target argmax_print

While there are some warnings, it compiles successfully, which means we have copied the plugin successfully and are ready to write our own code into it.

First, let’s write and compile the plugin, and then we will break down how it works.

  1. Rename the config to argmax_print_cfg.json:

    sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ cd ..
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ ls
    build  CMakeLists.txt  overlay.json  payload.cpp
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ mv overlay.json argmax_print_cfg.json

  2. Rewrite the contents argmax_print_cfg.json with:

    argmax_print_cfg.json
     1{
 2    "version": 0.1,
 3    "node_name": "argmax_print",
 4    "memory": {
 5        "cpu": 0,
 6        "next_cpu": 0
 7    },
 8    "system": {
 9        "out_buf_queue": 1,
10        "debug": 0,
11        "dump_data": 0
12    },
13    "buffers": {
14        "input": [
15        {
16            "name": "detess-dequant",
17            "size": 4000
18        }
19        ],
20        "output": {
21        "size": 4
22        }
23    }
24}

    Note

    Where the parameters:

    • ""node_name": Any name you want to give the instance of the plugin

    • ["memory"]["cpu"]: Means we are targeting A65 CPU memory space (0 -> A65 CPU, 1 -> EV74 CVU, 2 or 4 -> MLA)

    • ["memory"]["next_cpu"]: Means we are targeting the next plugin downstream to be A65 CPU memory space (0 -> A65 CPU, 1 -> EV74 CVU, 2 or 4 -> MLA)

    • ["system"]["out_buf_queue"]: Means how many buffers we are allocating to the output.

    • ["system"]["debug"]: 0 (false) or 1 (true) if we want debugging enabled.

    • ["system"]["dump_data"]: 0 (false) or 1 (true) if we want to dump the output buffer.

    • ["buffers"]["input"]["name"]: The input plugin’s name.

    • ["buffers"]["input"]["size"]: The input plugin’s buffer size. 1000 probabilities in float32 (4 bytes) is 4000 bytes.

    • ["buffers"]["output"]["size"]: The input plugin’s output. We are passing the index of the largest probability to the next plugin. In this case, one int32 value, or 4 bytes.


    You can run gst-inspect-1.0 --gst-plugin-path=/data/simaai/building_apps_palette/gstreamer/gst-plugins argmax_print for more information on the plugin itself.

  3. Re-write the payload.cpp with the following:

    payload.cpp
     1#include <aggregator/agg_template.h>
 2
 3// Custom functions
 4
 5// Function to perform argmax
 6int argmax(const float* buffer, size_t size) {
 7    // Use std::max_element to find the index of the largest value
 8    return std::distance(buffer, std::max_element(buffer, buffer + size));
 9}
10
11// Here's your init code
12UserContext::UserContext(nlohmann::json* json) : parser(json) {
13    std::cout << PLUGIN_STR_C " INIT" << std::endl;
14}
15
16// Here's your deinit code
17UserContext::~UserContext() {
18    std::cout << PLUGIN_STR_C " DEINIT" << std::endl;
19}
20
21// this code will be called on every new frame
22void UserContext::run(std::vector<Input> &input,
23                      std::span<uint8_t> output) {
24    // Extract the input probabilities
25    float * probabilities = reinterpret_cast<float*>(input[0].getData().data());
26    std::size_t probabilities_size = input[0].getDataSize();
27
28    // Perform the argmax operation
29    int max_idx = argmax(probabilities, probabilities_size);
30
31    // Print the index of the largest value
32    std::cout << "Index of largest value: " << max_idx << std::endl;
33    std::cout << "Largest value: " << probabilities[max_idx] << std::endl;
34
35    // Send the output out, but first ensure enough memory has been allocated for the output
36    if (output.size() >= sizeof(int)) {
37        // Copy the index into the output buffer (reinterpret int as bytes)
38        std::memcpy(output.data(), &max_idx, sizeof(int));
39    } else {
40        std::cerr << "Error: Output buffer is too small to store the index." << std::endl;
41    }
42}

  4. Make the build directory and compile

    sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ source /opt/poky/4.0.10/environment-setup-cortexa65-poky-linux
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ cd build
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ cmake .. && make clean && make
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build
    [ 50%] Building CXX object CMakeFiles/argmax_print.dir/payload.cpp.o
    [100%] Linking CXX shared library libargmax_print.so
    [100%] Built target argmax_print

  5. From the Palette docker container, copy the argmax_print_cfg.json and the newly built plugin libargmax_print.so to the MLSoC board:

    sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ scp libargmax_print.so sima@<IP address of MLSoC>:/home/sima/gst-plugins/
sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ scp ../argmax_print_cfg.json sima@<IP address of MLSoC>:/home/sima/resnet50_example_app/app_configs/

  6. Let’s update the previous run_pipeline.sh script to include our new plugin.

    #!/bin/bash

# Constants #
APP_DIR=/home/sima/resnet50_example_app
DATA_DIR="${APP_DIR}/data"
SIMA_PLUGINS_DIR="${APP_DIR}/../gst-plugins"
SAMPLE_IMAGE_SRC="${DATA_DIR}/golden_retriever_207_rgb.bin"
CONFIGS_DIR="${APP_DIR}/app_configs"
PREPROC_CVU_CONFIG_BIN="${CONFIGS_DIR}/genpreproc_200_cvu_cfg_app"
PREPROC_CVU_CONFIG_JSON="${CONFIGS_DIR}/genpreproc_200_cvu_cfg_params.json"
INFERENCE_MLA_CONFIG_JSON="${CONFIGS_DIR}/simaaiprocessmla_cfg_params.json"
DETESSDEQUANT_CVU_CONFIG_BIN="${CONFIGS_DIR}/detessdequant_201_cvu_cfg_app"
DETESSDEQUANT_CVU_CONFIG_JSON="${CONFIGS_DIR}/detessdequant_201_cvu_cfg_params.json"
ARGMAX_PRINT_CONFIG_JSON="${CONFIGS_DIR}/argmax_print_cfg.json"

# Remove any existing temporary files before running
rm /tmp/generic_preproc*.out /tmp/mla-*.out /tmp/detess-dequant*.out

# Run the configuration apps for generic_preproc and detessdequant
$PREPROC_CVU_CONFIG_BIN $PREPROC_CVU_CONFIG_JSON
$DETESSDEQUANT_CVU_CONFIG_BIN $DETESSDEQUANT_CVU_CONFIG_JSON

# Run the application
export LD_LIBRARY_PATH="${SIMA_PLUGINS_DIR}"
gst-launch-1.0 -v --gst-plugin-path="${SIMA_PLUGINS_DIR}" \
simaaisrc mem-target=1 node-name="my_image_src" location="${SAMPLE_IMAGE_SRC}" num-buffers=1 ! \
simaaiprocesscvu source-node-name="my_image_src" buffers-list="my_image_src" config="${PREPROC_CVU_CONFIG_JSON}" ! \
simaaiprocessmla config="${INFERENCE_MLA_CONFIG_JSON}" ! \
simaaiprocesscvu source-node-name="mla-resnet" buffers-list="mla-resnet" config="${DETESSDEQUANT_CVU_CONFIG_JSON}" ! \
argmax_print config="${ARGMAX_PRINT_CONFIG_JSON}" ! \
fakesink

  7. Finally let’s run the full end-to-end application!

    davinci:/data/simaai/building_apps_palette/gstreamer/resnet50_example_app$ sudo sh run_pipeline.sh
    Password:
    Completed SIMA_GENERIC_PREPROC graph configure

    Completed Graph Configure
    ** Message: 00:15:40.779: Num of chunks 1
    ** Message: 00:15:40.779: Buffer_name: my_image_src, num_of_chunks:1

    (gst-launch-1.0:30032): GLib-GObject-CRITICAL **: 00:15:40.790: g_pointer_type_register_static: assertion 'g_type_from_name (name) == 0' failed

    (gst-launch-1.0:30032): GLib-GObject-CRITICAL **: 00:15:40.790: g_type_set_qdata: assertion 'node != NULL' failed

    (gst-launch-1.0:30032): GLib-GObject-CRITICAL **: 00:15:40.791: g_pointer_type_register_static: assertion 'g_type_from_name (name) == 0' failed

    (gst-launch-1.0:30032): GLib-GObject-CRITICAL **: 00:15:40.791: g_type_set_qdata: assertion 'node != NULL' failed
    ** Message: 00:15:40.791: Num of chunks 1
    ** Message: 00:15:40.792: Buffer_name: mla-resnet, num_of_chunks:1
    Setting pipeline to PAUSED ...
    Argmax_print INIT
    ** Message: 00:15:40.802: Initialize dispatcher
    ** Message: 00:15:40.803: handle: 0x83ff95b0, 0xffff83ff95b0
    ** Message: 00:15:41.460: Loaded model from location /home/sima/resnet50_example_appmodels/quantized_resnet50_stage1_mla.lm, model:hdl: 0xaaaac0357c60
    ** Message: 00:15:41.465: Filename memalloc = /home/sima/resnet50_example_appdata/golden_retriever_207_rgb.bin
    Pipeline is PREROLLING ...
    Index of largest value: 207
    Largest value: 0.959188
    Pipeline is PREROLLED ...
    Setting pipeline to PLAYING ...
    Redistribute latency...
    New clock: GstSystemClock
    Got EOS from element "pipeline0".
    Execution ended after 0:00:00.001523003
    Setting pipeline to NULL ...
    Freeing pipeline ...

    Note

    If you notice on lines 27 and 28 we can see the plugins output as we would expect.

Conclusion and next steps

In this section, we:

  • Learned how to develop and build a simple custom plugin using SiMa.ai’s custom plugin template (simple_overlay)

  • Successfully finished our first end-to-end application!

Now that we have developed and debugged our end-to-end pipeline, we are are ready to package it into an ML Pipeline Package (mpk). This will allow you to conveniently deploy the application on any SiMa.ai MLSoC.