.. _developing_gstreamer_app_gstreamer_argmaxprint:

Step 4: Develop, test and verify output of ``argmax_print`` custom plugin
#########################################################################

.. image:: media/resnet50_application_simaaisrc_argmax_print.jpg 
    :align: center
    :scale: 30%

|

As a final step, we need to perform ``argmax`` to select the highest probability.
The output from the network is an array of 1000 probabilities. Thus, we should expect an output of 1000 fp32 values, or 4000 bytes total (fp32 => 4 bytes).

To do so, we will create a custom plugin where we will:

#. Take in the buffer from the ``simaaiprocesscvu`` (running the :ref:`ev74_graph_201_sima_detess_dequant` CVU graph) plugin.
#. Perform the ``argmax`` operation on it.
#. Print the output class index to a console.

Creating a custom Plugin
========================

On your host machine's Palette docker install, go to SiMa's pluginzoo directory, copy the ``simple_overlay`` plugin example and compile as a test:

.. code-block:: console

    sima-user@docker-image-id:~$ source /opt/poky/4.0.10/environment-setup-cortexa65-poky-linux
    sima-user@docker-image-id:~$ cd /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst
    sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst$ cp -r examples/opencv/simple_overlay ./argmax_print
    sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst$ cd argmax_print/
    sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ mkdir build && cd build
    sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ cmake ..
        -- Configuring done
        -- Generating done
        -- Build files have been written to: /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build
    sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ make
        [ 50%] Building CXX object CMakeFiles/argmax_print.dir/payload.cpp.o
        /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/payload.cpp: In member function ‘void UserContext::run(std::vector<Input>&, std::span<unsigned char>)’:
        /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/payload.cpp:57:71: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
        57 |     cv::imencode(".jpg", image, reinterpret_cast<std::vector<uchar>&>(output));
            |                                                                       ^~~~~~
        [100%] Linking CXX shared library libargmax_print.so
        [100%] Built target argmax_print

While there are some warnings, it compiles successfully, which means we have copied the plugin successfully and are ready to write our own code into it.

First, let's write and compile the plugin, and then we will break down how it works.

#. Rename the config to ``argmax_print_cfg.json``:

    .. code-block:: console

        sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ cd ..
        sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ ls
            build  CMakeLists.txt  overlay.json  payload.cpp
        sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ mv overlay.json argmax_print_cfg.json

#. Rewrite the contents ``argmax_print_cfg.json`` with:

    .. code-block:: json
        :caption: argmax_print_cfg.json
        :linenos:

        {
            "version": 0.1,
            "node_name": "argmax_print",
            "memory": {
                "cpu": 0,
                "next_cpu": 0
            },
            "system": {
                "out_buf_queue": 1,
                "debug": 0,
                "dump_data": 0
            },
            "buffers": {
                "input": [
                {
                    "name": "detess-dequant",
                    "size": 4000
                }
                ],
                "output": {
                "size": 4
                }
            }
        }

    .. note::
        Where the parameters:
        
        * ``""node_name"``: Any name you want to give the instance of the plugin
        * ``["memory"]["cpu"]``: Means we are targeting A65 CPU memory space (``0`` -> A65 CPU, ``1`` -> EV74 CVU, ``2`` or ``4`` -> MLA)
        * ``["memory"]["next_cpu"]``: Means we are targeting the next plugin downstream to be A65 CPU memory space (``0`` -> A65 CPU, ``1`` -> EV74 CVU, ``2`` or ``4`` -> MLA)
        * ``["system"]["out_buf_queue"]``: Means how many buffers we are allocating to the output.
        * ``["system"]["debug"]``: 0 (false) or 1 (true) if we want debugging enabled.
        * ``["system"]["dump_data"]``: 0 (false) or 1 (true) if we want to dump the output buffer.
        * ``["buffers"]["input"]["name"]``: The input plugin's name.
        * ``["buffers"]["input"]["size"]``: The input plugin's buffer size. 1000 probabilities in float32 (4 bytes) is 4000 bytes.
        * ``["buffers"]["output"]["size"]``: The input plugin's output. We are passing the index of the largest probability to the next plugin. In this case, one int32 value, or 4 bytes.
        
        |
        
        You can run ``gst-inspect-1.0 --gst-plugin-path=/data/simaai/building_apps_palette/gstreamer/gst-plugins argmax_print`` for more information on the plugin itself.

#. Re-write the ``payload.cpp`` with the following:

    .. code-block:: cpp
        :caption: payload.cpp
        :linenos:

        #include <aggregator/agg_template.h>

        // Custom functions

        // Function to perform argmax
        int argmax(const float* buffer, size_t size) {
            // Use std::max_element to find the index of the largest value
            return std::distance(buffer, std::max_element(buffer, buffer + size));
        }

        // Here's your init code
        UserContext::UserContext(nlohmann::json* json) : parser(json) {
            std::cout << PLUGIN_STR_C " INIT" << std::endl;
        }

        // Here's your deinit code
        UserContext::~UserContext() {
            std::cout << PLUGIN_STR_C " DEINIT" << std::endl;
        }

        // this code will be called on every new frame
        void UserContext::run(std::vector<Input> &input,
                              std::span<uint8_t> output) {
            // Extract the input probabilities
            float * probabilities = reinterpret_cast<float*>(input[0].getData().data());
            std::size_t probabilities_size = input[0].getDataSize();

            // Perform the argmax operation
            int max_idx = argmax(probabilities, probabilities_size);

            // Print the index of the largest value
            std::cout << "Index of largest value: " << max_idx << std::endl;
            std::cout << "Largest value: " << probabilities[max_idx] << std::endl;

            // Send the output out, but first ensure enough memory has been allocated for the output
            if (output.size() >= sizeof(int)) {
                // Copy the index into the output buffer (reinterpret int as bytes)
                std::memcpy(output.data(), &max_idx, sizeof(int));
            } else {
                std::cerr << "Error: Output buffer is too small to store the index." << std::endl;
            }
        }  

#. Make the build directory and compile

    .. code-block:: console

        sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ source /opt/poky/4.0.10/environment-setup-cortexa65-poky-linux
        sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print$ cd build
        sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ cmake .. && make clean && make
            -- Configuring done
            -- Generating done
            -- Build files have been written to: /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build
            [ 50%] Building CXX object CMakeFiles/argmax_print.dir/payload.cpp.o
            [100%] Linking CXX shared library libargmax_print.so
            [100%] Built target argmax_print


#. From the Palette docker container, copy the ``argmax_print_cfg.json`` and the newly built plugin ``libargmax_print.so`` to the MLSoC board:

    .. code-block:: console

        sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ scp libargmax_print.so sima@<IP address of MLSoC>:/home/sima/gst-plugins/
        sima-user@docker-image-id:/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/build$ scp ../argmax_print_cfg.json sima@<IP address of MLSoC>:/home/sima/resnet50_example_app/app_configs/

#. Let's update the previous ``run_pipeline.sh`` script to include our new plugin.

    .. code-block:: bash

        #!/bin/bash

        # Constants #
        APP_DIR=/home/sima/resnet50_example_app
        DATA_DIR="${APP_DIR}/data"
        SIMA_PLUGINS_DIR="${APP_DIR}/../gst-plugins"
        SAMPLE_IMAGE_SRC="${DATA_DIR}/golden_retriever_207_rgb.bin"
        CONFIGS_DIR="${APP_DIR}/app_configs"
        PREPROC_CVU_CONFIG_BIN="${CONFIGS_DIR}/genpreproc_200_cvu_cfg_app"
        PREPROC_CVU_CONFIG_JSON="${CONFIGS_DIR}/genpreproc_200_cvu_cfg_params.json"
        INFERENCE_MLA_CONFIG_JSON="${CONFIGS_DIR}/simaaiprocessmla_cfg_params.json"
        DETESSDEQUANT_CVU_CONFIG_BIN="${CONFIGS_DIR}/detessdequant_201_cvu_cfg_app"
        DETESSDEQUANT_CVU_CONFIG_JSON="${CONFIGS_DIR}/detessdequant_201_cvu_cfg_params.json"
        ARGMAX_PRINT_CONFIG_JSON="${CONFIGS_DIR}/argmax_print_cfg.json"

        # Remove any existing temporary files before running
        rm /tmp/generic_preproc*.out /tmp/mla-*.out /tmp/detess-dequant*.out

        # Run the configuration apps for generic_preproc and detessdequant
        $PREPROC_CVU_CONFIG_BIN $PREPROC_CVU_CONFIG_JSON
        $DETESSDEQUANT_CVU_CONFIG_BIN $DETESSDEQUANT_CVU_CONFIG_JSON

        # Run the application
        export LD_LIBRARY_PATH="${SIMA_PLUGINS_DIR}"
        gst-launch-1.0 -v --gst-plugin-path="${SIMA_PLUGINS_DIR}" \
        simaaisrc mem-target=1 node-name="my_image_src" location="${SAMPLE_IMAGE_SRC}" num-buffers=1 ! \
        simaaiprocesscvu source-node-name="my_image_src" buffers-list="my_image_src" config="${PREPROC_CVU_CONFIG_JSON}" ! \
        simaaiprocessmla config="${INFERENCE_MLA_CONFIG_JSON}" ! \
        simaaiprocesscvu source-node-name="mla-resnet" buffers-list="mla-resnet" config="${DETESSDEQUANT_CVU_CONFIG_JSON}" ! \
        argmax_print config="${ARGMAX_PRINT_CONFIG_JSON}" ! \
        fakesink

#. Finally let's run the full end-to-end application!

    .. code-block:: console

        davinci:/data/simaai/building_apps_palette/gstreamer/resnet50_example_app$ sudo sh run_pipeline.sh
            Password: 
            Completed SIMA_GENERIC_PREPROC graph configure 

            Completed Graph Configure 
            ** Message: 00:15:40.779: Num of chunks 1
            ** Message: 00:15:40.779: Buffer_name: my_image_src, num_of_chunks:1

            (gst-launch-1.0:30032): GLib-GObject-CRITICAL **: 00:15:40.790: g_pointer_type_register_static: assertion 'g_type_from_name (name) == 0' failed

            (gst-launch-1.0:30032): GLib-GObject-CRITICAL **: 00:15:40.790: g_type_set_qdata: assertion 'node != NULL' failed

            (gst-launch-1.0:30032): GLib-GObject-CRITICAL **: 00:15:40.791: g_pointer_type_register_static: assertion 'g_type_from_name (name) == 0' failed

            (gst-launch-1.0:30032): GLib-GObject-CRITICAL **: 00:15:40.791: g_type_set_qdata: assertion 'node != NULL' failed
            ** Message: 00:15:40.791: Num of chunks 1
            ** Message: 00:15:40.792: Buffer_name: mla-resnet, num_of_chunks:1
            Setting pipeline to PAUSED ...
            Argmax_print INIT
            ** Message: 00:15:40.802: Initialize dispatcher
            ** Message: 00:15:40.803: handle: 0x83ff95b0, 0xffff83ff95b0
            ** Message: 00:15:41.460: Loaded model from location /home/sima/resnet50_example_appmodels/quantized_resnet50_stage1_mla.lm, model:hdl: 0xaaaac0357c60
            ** Message: 00:15:41.465: Filename memalloc = /home/sima/resnet50_example_appdata/golden_retriever_207_rgb.bin
            Pipeline is PREROLLING ...
            Index of largest value: 207
            Largest value: 0.959188
            Pipeline is PREROLLED ...
            Setting pipeline to PLAYING ...
            Redistribute latency...
            New clock: GstSystemClock
            Got EOS from element "pipeline0".
            Execution ended after 0:00:00.001523003
            Setting pipeline to NULL ...
            Freeing pipeline ...

    .. note::

        If you notice on lines ``27`` and ``28`` we can see the plugins output as we would expect.

Conclusion and next steps
=========================

In this section, we: 

    * Learned how to develop and build a simple custom plugin using SiMa.ai's custom plugin template (simple_overlay)
    * Successfully finished our first end-to-end application!

Now that we have developed and debugged our end-to-end pipeline, we are are ready to package it into an ML Pipeline Package (``mpk``).
This will allow you to conveniently deploy the application on any SiMa.ai MLSoC.