.. _developing_mpk_app: .. |mpk_pipeline| replace:: MPK Application .. |plugin_base| replace:: ``/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base`` .. |app_name| replace:: ResNet50 Sample App Developing the |mpk_pipeline| ############################# An mpk application is used to conveniently package and deploy an application to MLSoC target devices using MPK CLI tools included with Palette software. An |mpk_pipeline| is just a package that executes a GStreamer application at runtime. This section describes how we can create an end to end working ML Pipeline Package (``mpk``) in a step-by-step manner. We will finish our ResNet50 application by packaging it up into an ``mpk`` and deploying it on the board. To create an application ``mpk`` for the MLSoC: * **Test and debug** an application using ``gst-launch-1.0`` - or start from an existing example application and edit. * For more information refer to the :ref:`developing_gstreamer_app` guide to get started. * **Define a project structure** along with necessary configuration files to define the application. The compile and package using ``mpk create`` CLI tool. * **Deploy the application** using ``mpk deploy`` CLI tool to deploy the created ``mpk`` application to the MLSoC board. .. important:: The creation of an |mpk_pipeline| happens within the **Palette docker container**. All plugins and resources are cross-compiled and packaged into a resulting ``.mpk`` package with the use of the ``mpk create`` command as we will see later in this guide. .. _developing_mpk_folder_structure: Folder Structure ================ Creating an |mpk_pipeline| requires a specific folder structure. Every folder serves a purpose, let's discuss them in detail. The folder tree is as follows: .. code-block:: console application_name ├── plugins ├── .project │ └── pluginsInfo.json ├── resources ├── core └── application.json ``application_name`` -------------------- * This is the master folder for the target application we are developing and holds all necessary resources inside. ``plugins`` ----------- * This directory holds all the different :ref:`plugins` plugins being used in the pipeline. * Plugins needed for our application are located in |plugin_base| as we have seen during our GStreamer development and debugging. ``.project`` ------------ * This is a hidden directory internally required by the ``mpk create`` command while doing a compilation of plugins available in the |plugin_base|. * This directory holds a very important file named ``pluginsInfo.json``. This file contains all the plugins' list which are being used in the |mpk_pipeline|. ``resources`` ------------- * This directory is keeping the |dependent_app| required to load different EV74 graphs. It also holds any necessary dependencies that need to be executed before actually running the pipeline on target device. ``core`` -------- * This directory holds the ``dispatcher`` present under ``core`` directory at |plugin_base|. * This dispatcher is necessary for dispatching data on ``MLA`` and is required by :ref:`plugin_simaaiprocessmla`. ``application.json`` -------------------- * This is a complete description of the application in JSON format. This file holds various information related to pipeline's name, plugins, gstreamer command, etc. Pre-Requisites ============== Before creating an |mpk_pipeline|, we should: #. Have a reference of the application we are developing. In our case, this will be the :ref:`developing_gstreamer_app_reference_app`. #. Have debugged the application using GStreamer `gst-launch` strings to ensure everything is working as expected. #. Should have the latest version of Palette installed with the corresponding firmware on the MLSoC. Let's remind ourselves of the application we will package using the an ``mpk``: The application logic: .. image:: ../developing_gstreamer_apps/media/resnet50_application_with_images.jpg Mapping it to the MLSoC: .. image:: ../developing_gstreamer_apps/media/resnet50_application_hw.jpg | Step-by-Step Guide ================== Create the directory structure ------------------------------ As discussed above in the previous section, let's create the folder structure for our |app_name|. #. Create project directory structure .. code-block:: console sima-user@docker-image-id:~$ cd /home/docker/sima-cli sima-user@docker-image-id:/home/docker/sima-cli$ mkdir resnet50_app && cd resnet50_app sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mkdir plugins .project resources && touch application.json .project/pluginsInfo.json sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 . ├── application.json ├── plugins ├── .project │ └── pluginsInfo.json └── resources #. Add some boilerplate in ``.project/pluginsInfo.json`` running the following command under ``/home/docker/sima-cli/resnet50_app``: .. code-block:: bash echo '{ "pluginsInfo": [] }' > .project/pluginsInfo.json #. Add some boilerplate in ``application.json`` running the following command under ``/home/docker/sima-cli/resnet50_app``: .. code-block:: bash echo '{ "name": "resnet50_sample_name", "appId": "resnet50_sample_id", "sequence": 1, "pipelines": [ { "name": "resnet50_sample_pipeline_name", "sequence": 1, "plugins": [], "connections": [], "resources": [], "dependencies": [] } ] }' > application.json .. note:: Some notable parameters: * ``["name"]``: Name of the application as it will be installed on the MLSoC device. * ``["appID"]``: Unique ID for the application to differentiate from other applications. * ``["pipelines"]``: Where the pipeline definition will go. Only support one pipeline at the moment. * ``["resources"]``: Any resources that the application depends on (for example: CVU configuration applications). * ``["dependencies"]``: Any programs or commands that the application has to run at runtime. Install ``uuidgen`` ------------------- #. To create GUID for every plugin, one needs to use ``uuidgen`` tool which needs to be installed additionally if not present. .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ sudo apt-get update && sudo apt-get install uuid-runtime #. For every plugin specification, generate the GUID as shown below: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen 1627c73c-bf68-49af-b9d4-89fb6c13dad5 Adding Plugins -------------- How to add a new plugin ^^^^^^^^^^^^^^^^^^^^^^^ Now that we have the skeleton structure of creating an |mpk_pipeline|, let's begin adding the plugins of the pipeline. **Overall, the process of adding each plugin is as follows:** #. Copy the plugin from |plugin_base| to our plugins directory ``/home/docker/sima-cli/resnet50_app/plugins/`` * Bring the sources of the plugin into the application for the ``mpk create`` framework to cross-compile the plugin. * If using the same plugin more than once, no need to copy it multiple times. #. Create a ``cfg/`` directory for your plugin configurations inside of the copied plugin folder (not always necessary) * This is where we will store any JSONs or configuration files associated with this plugin * For the ``simaaiprocesscvu`` to run :ref:`cvu_graphs` on the EV74 CVU, this is also where we will set up the ``configuration application`` #. Create the configuration file inside the ``cfg/`` directory (not always necessary) * Location to store configuration files for the plugin #. Update the ``.project/pluginsInfo.json`` with the path to the new plugin along with a new generated ``uuidgen`` id * This creates a mapping of our plugin directory to a unique ID recognizable by the ``mpk create`` framework #. Update the ``application.json`` with the new plugin's ``name``, ``pluginGid`` (same as previous step), and ``[resources][configs]`` * This step associates a created plugin and it's configuration to the overall application. Plugins for our ResNet50 pipeline ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If we recall from :ref:`developing_gstreamer_app_gstreamer_app_overview`, we will need the following plugins as part of our application: ``simaaisrc`` -> ``simaaiprocesscvu`` -> ``simaaiprocessmla`` -> ``simaaiprocesscvu`` -> ``argmax_print`` -> ``fakesink`` Where: * ``argmax_print`` is our custom plugin * ``fakesink`` is an open source GStreamer plugin that we do not have to manually compile or copy over to our project structure. Adding SiMa GStreamer Plugins ----------------------------- Adding ``simaaisrc`` ^^^^^^^^^^^^^^^^^^^^ Description *********** In this step we will be adding the file source to our pipeline as it was tested in: * :ref:`developing_gstreamer_app_gstreamer_inputsrc_a65` Steps ***** #. Copy the plugin from |plugin_base| to ``/home/docker/sima-cli/resnet50_app/plugins/``: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/simaaisrc plugins/ sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 . ├── application.json ├── plugins │ └── simaaisrc │ ├── CMakeLists.txt │ ├── gstsimaaisrc.cc │ ├── gstsimaaisrc.h │ └── README.md ├── .project │ └── pluginsInfo.json └── resources .. note:: If you have previously compiled the plugin for testing, you will also see a ``build/`` directory in the ``plugins/simaaisrc/`` directory. #. Create a ``cfg/`` directory: * SKIP. This plugin does not require any configuration files as all configurations will be done via command line parameters later. #. Create the configuration file inside the ``cfg/`` directory: * SKIP. This plugin does not require any configuration files as all configurations will be done via command line parameters later. #. Update the ``.project/pluginsInfo.json`` with the path to the new plugin along with a new generated ``uuidgen`` id. * Generate a new ``uuid`` for this plugin: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen 0c255b9a-e5a8-4346-993f-e3527cc6d4e3 * Update the ``.project/pluginsInfo.json`` to: .. code-block:: json :linenos: { "pluginsInfo" : [ { "gid" : "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "path" : "plugins/simaaisrc" } ] } #. Update the ``application.json`` with the new plugin's ``name``, ``pluginGid`` (same as previous step), and ``[resources][configs]``: .. code-block:: json :caption: application.json :linenos: { "name": "resnet50_sample_name", "appId": "resnet50_sample_id", "sequence": 1, "pipelines": [ { "name": "resnet50_sample_pipeline_name", "sequence": 1, "plugins": [ { "name": "simaaisrc", "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "resources": { "configs": [] } } ], "connections": [], "resources": [], "dependencies": [] } ] } Adding ``simaaiprocesscvu`` (preprocessing) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description *********** In this step we will be adding the ResNet50 preprocessing to our pipeline as it was tested in: * :ref:`developing_gstreamer_app_gstreamer_preproc_cvu` Steps ***** #. Copy the plugin from |plugin_base| to ``/home/docker/sima-cli/resnet50_app/plugins/``: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/processcvu/ plugins/ sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 . ├── application.json ├── plugins │ ├── processcvu │ │ ├── CMakeLists.txt │ │ ├── gstsimaaiprocesscvu.cpp │ │ ├── gstsimaaiprocesscvu.h │ │ └── README.md │ └── simaaisrc │ ├── CMakeLists.txt │ ├── gstsimaaisrc.cc │ ├── gstsimaaisrc.h │ └── README.md ├── .project │ └── pluginsInfo.json └── resources #. Create a ``cfg/`` directory: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mkdir plugins/processcvu/cfg sima-user@docker-image-ide:/home/docker/sima-cli/resnet50_app$ tree -aL 3 . ├── application.json ├── plugins │ ├── processcvu │ │ ├── cfg │ │ ├── CMakeLists.txt │ │ ├── gstsimaaiprocesscvu.cpp │ │ ├── gstsimaaiprocesscvu.h │ │ └── README.md │ └── simaaisrc │ ├── CMakeLists.txt │ ├── gstsimaaisrc.cc │ ├── gstsimaaisrc.h │ └── README.md ├── .project │ └── pluginsInfo.json └── resources #. Create the configuration file inside the ``cfg/`` directory: * Copy the JSON ``genpreproc_200_cvu_cfg_params.json`` configuration file to this location or re-create the file. * To copy it: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp /home/docker/sima-cli/ev74_cgfs/sima_generic_preproc/genpreproc_200_cvu_cfg_params.json plugins/processcvu/cfg/ .. note:: Copying the configuration file depends on having followed the pre-requisite guide: :ref:`developing_gstreamer_app_gstreamer_preproc_cvu` found under :ref:`developing_gstreamer_app` * To recreate it run the following command under ``/home/docker/sima-cli/resnet50_app``: .. code-block:: bash echo '{ "version": 0.1, "node_name": "generic_preproc", "simaai__params": { "params": 15, "cpu": 1, "next_cpu": 4, "no_of_outbuf": 1, "ibufname": "null", "graph_id": 200, "img_width": 1920, "img_height": 1080, "input_width": 1920, "input_height": 1080, "output_width": 224, "output_height": 224, "scaled_width": 224, "scaled_height": 224, "batch_size": 1, "normalize": 1, "rgb_interleaved": 1, "aspect_ratio": 0, "tile_width": 32, "tile_height": 16, "input_depth": 3, "output_depth": 3, "quant_zp": -14, "quant_scale": 53.59502566491281, "mean_r": 0.406, "mean_g": 0.456, "mean_b": 0.485, "std_dev_r": 0.225, "std_dev_g": 0.224, "std_dev_b": 0.229, "input_type": 2, "output_type": 0, "scaling_type": 3, "offset": 150528, "padding_type": 4, "input_stride": 0, "output_stride": 0, "output_dtype": 0, "debug": 0, "out_sz": 301056, "dump_data": 1 } }' > plugins/processcvu/cfg/genpreproc_200_cvu_cfg_params.json .. warning:: You should change the ``dump_data`` parameter to ``0`` when deploying the final application. We have it set to ``1`` currently to ensure we can test when we deploy that it works as expected. * The new directory structure should look like this: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 4 . ├── application.json ├── plugins │ ├── processcvu │ │ ├── cfg │ │ │ └── genpreproc_200_cvu_cfg_params.json │ │ ├── CMakeLists.txt │ │ ├── gstsimaaiprocesscvu.cpp │ │ ├── gstsimaaiprocesscvu.h │ │ └── README.md │ └── simaaisrc │ ├── CMakeLists.txt │ ├── gstsimaaisrc.cc │ ├── gstsimaaisrc.h │ └── README.md ├── .project │ └── pluginsInfo.json └── resources * Add the CVU configuration application for this plugin from: :ref:`developing_gstreamer_app_gstreamer_preproc_cvu`. We will first copy the compiled binary for the application and then we will add a small shell script to run the app at runtime. * Copy the pre-compiled application to the ``resources`` folder: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp /home/docker/sima-cli/ev74_cgfs/sima_generic_preproc/build/genpreproc_200_cvu_cfg_app resources/ .. note:: Copying the configuration app depends on having followed the pre-requisite guide: :ref:`developing_gstreamer_app_gstreamer_preproc_cvu` found under :ref:`developing_gstreamer_app` * Create a shell script: This shell script will run before the GStreamer pipeline kicks off. To generate the file run the following command under ``/home/docker/sima-cli/resnet50_app``: .. code-block:: bash echo ' #!/bin/sh APP_DIR="/data/simaai/applications" APP_NAME="resnet50_sample_name" ETC_PATH="${APP_DIR}/${APP_NAME}/etc" PREPROC_CVU_CONFIG_BIN="genpreproc_200_cvu_cfg_app" PREPROC_CVU_CONFIG_JSON="genpreproc_200_cvu_cfg_params.json" # Execute preprocessing config ${ETC_PATH}/${PREPROC_CVU_CONFIG_BIN} ${ETC_PATH}/${PREPROC_CVU_CONFIG_JSON} ' > resources/run_cvu_cfgs.sh .. note:: This script is similar to the ones we created in the :ref:`developing_gstreamer_app_gstreamer_preproc_cvu` guide, with two notable differences: * The directories are specific to where an** ``mpk`` application is installed. * All ``mpk`` applications are installed under ``/data/simaai/applications/``. This application will be installed under ``/data/simaai/applications/resnet50_sample_name/``. * The script purely invokes the CVU configuration application, and does not call ``gst-launch-1.0``. * The new directory structure should look like this: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 4 . ├── application.json ├── plugins │ ├── processcvu │ │ ├── cfg │ │ │ └── genpreproc_200_cvu_cfg_params.json │ │ ├── CMakeLists.txt │ │ ├── gstsimaaiprocesscvu.cpp │ │ ├── gstsimaaiprocesscvu.h │ │ └── README.md │ └── simaaisrc │ ├── CMakeLists.txt │ ├── gstsimaaisrc.cc │ ├── gstsimaaisrc.h │ └── README.md ├── .project │ └── pluginsInfo.json └── resources ├── genpreproc_200_cvu_cfg_app └── run_cvu_cfgs.sh #. Update the ``.project/pluginsInfo.json`` with the path to the new plugin along with a new generated ``uuidgen`` id. * Generate a new ``uuid`` for this plugin: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen 09b09793-e1de-4881-9fd8-7674b948864a * Update the ``.project/pluginsInfo.json`` to: .. code-block:: json :linenos: { "pluginsInfo" : [ { "gid" : "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "path" : "plugins/simaaisrc" }, { "gid" : "09b09793-e1de-4881-9fd8-7674b948864a", "path" : "plugins/processcvu" } ] } #. Update the ``application.json`` with the new plugin's ``name``, ``pluginGid`` (same as previous step), and ``[resources][configs]``: .. code-block:: json :caption: application.json :linenos: { "name": "resnet50_sample_name", "appId": "resnet50_sample_id", "sequence": 1, "pipelines": [ { "name": "resnet50_sample_pipeline_name", "sequence": 1, "plugins": [ { "name": "simaaisrc", "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "resources": { "configs": [] } }, { "name": "processcvu", "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a", "resources": { "configs": [ "cfg/genpreproc_200_cvu_cfg_params.json" ] } } ], "connections": [], "resources": [ { "name": "genpreproc_200_cvu_cfg_app", "location": "resources/genpreproc_200_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "run_cvu_cfgs_sh", "location": "resources/run_cvu_cfgs.sh", "destination": { "provider": "RPM", "type": "bin" } } ], "dependencies": [ { "name": "run_cvu_cfgs_sh", "location": "/data/simaai/applications/resnet50_sample_name/bin/", "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh" } ] } ] } .. note:: We have made a large number of updates to the ``application.json`` so let us review the changes: * Added ``["pipelines"]["plugins"]["name"]: "processcvu"``: * Just like we did before, we are making our application aware of our new ``processcvu`` plugin we want to use. * Added ``["pipelines"]["plugins"]["resources"]["configs"]: "cfg/genpreproc_200_cvu_cfg_params.json"`` * Here we are providing the JSON configuration file that is needed to correctly initialize the plugin with the correct parameters. * Added ``["pipelines"]["resources"]["name"]: "genpreproc_200_cvu_cfg_app"``: * This makes our preprocessing CVU configuration binary available to our application. * Added ``["pipelines"]["resources"]["name"]: "run_cvu_cfgs_sh"``: * This makes our preprocessing CVU configuration shell script available to our application. * Added ``["pipelines"]["dependencies"]["name"]: "run_cvu_cfgs_sh"``: * This ensures that the application will run the ``run_cvu_cfgs_sh`` before executing the GStreamer application as specified by the ``["pipelines"]["dependencies"]["command"]`` field under ``["pipelines"]["dependencies"][0]`` (``run_cvu_cfgs_sh``). Adding ``simaaiprocessmla`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description *********** In this step we will be adding the ResNet50 inferencing to our pipeline as it was tested in: * :ref:`developing_gstreamer_app_gstreamer_inference_mla` Steps ***** #. Copy the plugin from |plugin_base| to ``/home/docker/sima-cli/resnet50_app/plugins/``: .. note:: * For the ``simaaiprocessmla`` plugin, we also need to copy over: * ``/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/core`` to the root of our application directory as well. .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/processmla/ plugins/ sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/core . sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 2 . ├── application.json ├── core │ ├── allocator │ ├── CMakeLists.txt │ ├── dispatcher-lite │ └── simamm ├── plugins │ ├── processcvu │ ├── processmla │ └── simaaisrc ├── .project │ └── pluginsInfo.json └── resources ├── genpreproc_200_cvu_cfg_app └── run_cvu_cfgs.sh #. Create a ``cfg/`` and ``res/`` directory: .. note:: * For the ``simaaiprocessmla`` plugin, we also need to create a ``res/`` directory: * We need a resource specifically for the plugin (the compiled model ``lm``), so we will create a resource directory ``res`` inside of our plugin as well. * Why not place ``lm`` model inside the ``/home/docker/sima-cli/resnet50_app/resources``? * Because that is reserved for application level resources, not resources needed for a specific plugin. * Create the directories: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mkdir plugins/processmla/cfg plugins/processmla/res sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 plugins/ plugins/ ├── processcvu │ ├── cfg │ │ └── genpreproc_200_cvu_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocesscvu.cpp │ ├── gstsimaaiprocesscvu.h │ └── README.md ├── processmla │ ├── cfg │ ├── CMakeLists.txt │ ├── gstsimaaiprocessmlacommon.h │ ├── gstsimaaiprocessmla.cpp │ ├── gstsimaaiprocessmla.h │ ├── gstsimaaiprocessmlastandalone.cpp │ ├── README.md │ └── res └── simaaisrc ├── CMakeLists.txt ├── gstsimaaisrc.cc ├── gstsimaaisrc.h └── README.md * Copy the `lm` model to the new ``res`` directory: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp /compiled_resnet50/quantized_resnet50_mpk/quantized_resnet50_stage1_mla.lm plugins/processmla/res/ sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 plugins/ plugins/ ├── processcvu │ ├── cfg │ │ └── genpreproc_200_cvu_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocesscvu.cpp │ ├── gstsimaaiprocesscvu.h │ └── README.md ├── processmla │ ├── cfg │ ├── CMakeLists.txt │ ├── gstsimaaiprocessmlacommon.h │ ├── gstsimaaiprocessmla.cpp │ ├── gstsimaaiprocessmla.h │ ├── gstsimaaiprocessmlastandalone.cpp │ ├── README.md │ └── res │ └── quantized_resnet50_stage1_mla.lm └── simaaisrc ├── CMakeLists.txt ├── gstsimaaisrc.cc ├── gstsimaaisrc.h └── README.md .. note:: The ``lm`` file is retrieved by extracting the contents of the ``tar.gz`` output of the ModelSDK. #. Create the configuration file inside the ``cfg/`` directory: * To recreate the ``simaaiprocessmla`` configuration file, run the following command under ``/home/docker/sima-cli/resnet50_app``: .. code-block:: bash echo '{ "version" : 0.1, "node_name" : "mla-resnet", "simaai__params" : { "params" : 15, "index" : 1, "cpu" : 4, "next_cpu" : 1, "out_sz" : 1008, "no_of_outbuf" : 1, "batch_size" : 1, "batch_sz_model" : 1, "in_tensor_sz": 0, "out_tensor_sz": 0, "ibufname" : "generic_preproc", "model_path" : "/home/sima/resnet50_example_app/models/quantized_resnet50_stage1_mla.lm", "debug" : 0, "dump_data": 1 } }' > plugins/processmla/cfg/simaaiprocessmla_cfg_params.json .. warning:: You should change the ``dump_data`` parameter to ``0`` when deploying the final application. We have it set to ``1`` currently to ensure we can test when we deploy that it works as expected. * The directory structure should now look like this: .. code-block:: sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 plugins/ plugins/ ├── processcvu │ ├── cfg │ │ └── genpreproc_200_cvu_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocesscvu.cpp │ ├── gstsimaaiprocesscvu.h │ └── README.md ├── processmla │ ├── cfg │ │ └── simaaiprocessmla_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocessmlacommon.h │ ├── gstsimaaiprocessmla.cpp │ ├── gstsimaaiprocessmla.h │ ├── gstsimaaiprocessmlastandalone.cpp │ ├── README.md │ └── res │ └── quantized_resnet50_stage1_mla.lm └── simaaisrc ├── CMakeLists.txt ├── gstsimaaisrc.cc ├── gstsimaaisrc.h └── README.md #. Update the ``.project/pluginsInfo.json`` with the path to the new plugin along with a new generated ``uuidgen`` id. * Generate a new ``uuid`` for this plugin: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen c086febf-7f33-447d-a4ed-96be718de36d * Update the ``.project/pluginsInfo.json`` to: .. code-block:: json :linenos: { "pluginsInfo" : [ { "gid" : "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "path" : "plugins/simaaisrc" }, { "gid" : "09b09793-e1de-4881-9fd8-7674b948864a", "path" : "plugins/processcvu" }, { "gid" : "c086febf-7f33-447d-a4ed-96be718de36d", "path" : "plugins/processmla" } ] } #. Update the ``application.json`` with the new plugin's ``name``, ``pluginGid`` (same as previous step), and ``[resources][configs]``: .. code-block:: json :caption: application.json :linenos: { "name": "resnet50_sample_name", "appId": "resnet50_sample_id", "sequence": 1, "pipelines": [ { "name": "resnet50_sample_pipeline_name", "sequence": 1, "plugins": [ { "name": "simaaisrc", "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "resources": { "configs": [] } }, { "name": "processcvu", "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a", "resources": { "configs": [ "cfg/genpreproc_200_cvu_cfg_params.json" ] } }, { "name": "processmla", "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d", "resources": { "configs": [ "cfg/simaaiprocessmla_cfg_params.json" ], "lms": [ "res/quantized_resnet50_stage1_mla.lm" ] }, "dependencies": { "value": [ "dispatcher-lite.zip" ] } } ], "connections": [], "resources": [ { "name": "genpreproc_200_cvu_cfg_app", "location": "resources/genpreproc_200_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "run_cvu_cfgs_sh", "location": "resources/run_cvu_cfgs.sh", "destination": { "provider": "RPM", "type": "bin" } } ], "dependencies": [ { "name": "run_cvu_cfgs_sh", "location": "/data/simaai/applications/resnet50_sample_name/bin/", "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh" } ] } ] } .. note:: Let's review the changes to the ``application.json``: * Added ``["pipelines"]["plugins"]["name"]: "processmla"``: * Just like we did before, we are making our application aware of our new ``processmla`` plugin we want to use. * Added ``["pipelines"]["plugins"]["processmla"]["resources"]["configs"]``: * Here we are providing the JSON configuration file that is needed to correctly initialize the plugin with the correct parameters. * Added ``["pipelines"]["plugins"]["processmla"]["resources"]["lms"]``: * The ``processmla`` plugin depends on a compiled ``lm`` model to run. Here we specify the name and location of that model. * Added ``["pipelines"]["plugins"]["processmla"]["dependencies"]``: * The ``processmla`` plugin depends the dispatcher library. ``mpk create`` will create a zip and place it in the right location for us. Adding ``simaaiprocesscvu`` (postprocessing) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description *********** In this step we will be adding the ResNet50 preprocessing to our pipeline as it was tested in: * :ref:`developing_gstreamer_app_gstreamer_postproc_cvu` Steps ***** #. Copy the plugin from |plugin_base| to ``/home/docker/sima-cli/resnet50_app/plugins/``: * SKIP. We already copied the ``simaaiprocesscvu`` plugin earlier to our project, no need to do it again. #. Create a ``cfg/`` directory: * SKIP. We already created a ``cfg/`` directory inside the ``processcvu`` directory, no need to do it again. #. Create the configuration file inside the ``cfg/`` directory: * Copy the JSON ``detessdequant_201_cvu_cfg_params.json`` configuration file to this location or re-create the file. * To copy it: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ ccp /home/docker/sima-cli/ev74_cgfs/sima_detess_dequant/detessdequant_201_cvu_cfg_params.json plugins/processcvu/cfg/ .. note:: Copying the configuration file depends on having followed the pre-requisite guide: :ref:`developing_gstreamer_app_gstreamer_postproc_cvu` found under :ref:`developing_gstreamer_app` * To recreate it, run the following command under ``/home/docker/sima-cli/resnet50_app``: .. code-block:: bash echo '{ "version": 0.1, "node_name": "detess-dequant", "simaai__params": { "params": 15, "cpu": 1, "next_cpu": 0, "no_of_outbuf": 1, "ibufname": "", "graph_id": 201, "img_width": 1280, "img_height": 720, "num_tensors": 1, "input_width": [1], "input_height": [1], "input_depth": [1000], "slice_width": [1], "slice_height": [1], "slice_depth": [1000], "dq_scale": [255.02200010497842], "dq_zp": [-128], "data_type": [0], "fp16_out_en": [0], "output_format": [0], "debug": 0, "out_sz": 4000, "dump_data": 1 } }' > plugins/processcvu/cfg/detessdequant_201_cvu_cfg_params.json .. warning:: You should change the ``dump_data`` parameter to ``0`` when deploying the final application. We have it set to ``1`` currently to ensure we can test when we deploy that it works as expected. * The new directory structure should look like this: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 plugins/ plugins/ ├── processcvu │ ├── cfg │ │ ├── detessdequant_201_cvu_cfg_params.json │ │ └── genpreproc_200_cvu_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocesscvu.cpp │ ├── gstsimaaiprocesscvu.h │ └── README.md ├── processmla │ ├── cfg │ │ └── simaaiprocessmla_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocessmlacommon.h │ ├── gstsimaaiprocessmla.cpp │ ├── gstsimaaiprocessmla.h │ ├── gstsimaaiprocessmlastandalone.cpp │ ├── README.md │ └── res │ └── quantized_resnet50_stage1_mla.lm └── simaaisrc ├── CMakeLists.txt ├── gstsimaaisrc.cc ├── gstsimaaisrc.h └── README.md * Add the CVU configuration application for this plugin from: :ref:`developing_gstreamer_app_gstreamer_postproc_cvu`. We will first copy the compiled binary for the application and then we will add a small shell script to run the app at runtime. * Copy the pre-compiled application to the ``resources`` folder: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp /home/docker/sima-cli/ev74_cgfs/sima_detess_dequant/build/detessdequant_201_cvu_cfg_app resources/ .. note:: Copying the configuration app depends on having followed the pre-requisite guide: :ref:`developing_gstreamer_app_gstreamer_postproc_cvu` found under :ref:`developing_gstreamer_app` * Run the following command under ``/home/docker/sima-cli/resnet50_app`` to modify the shell script to add how to run the postprocessing CVU configuration application: .. code-block:: bash echo ' #!/bin/sh APP_DIR="/data/simaai/applications" APP_NAME="resnet50_sample_name" ETC_PATH="${APP_DIR}/${APP_NAME}/etc" PREPROC_CVU_CONFIG_BIN="genpreproc_200_cvu_cfg_app" PREPROC_CVU_CONFIG_JSON="genpreproc_200_cvu_cfg_params.json" DETESSDEQUANT_CVU_CONFIG_BIN="detessdequant_201_cvu_cfg_app" DETESSDEQUANT_CVU_CONFIG_JSON="detessdequant_201_cvu_cfg_params.json" # Execute preprocessing config ${ETC_PATH}/${PREPROC_CVU_CONFIG_BIN} ${ETC_PATH}/${PREPROC_CVU_CONFIG_JSON} # Execute postprocessing config ${ETC_PATH}/${DETESSDEQUANT_CVU_CONFIG_BIN} ${ETC_PATH}/${DETESSDEQUANT_CVU_CONFIG_JSON} ' > resources/run_cvu_cfgs.sh * The new directory structure should look like this: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 4 plugins plugins/ ├── processcvu │ ├── cfg │ │ ├── detessdequant_201_cvu_cfg_params.json │ │ └── genpreproc_200_cvu_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocesscvu.cpp │ ├── gstsimaaiprocesscvu.h │ └── README.md ├── processmla │ ├── cfg │ │ └── simaaiprocessmla_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocessmlacommon.h │ ├── gstsimaaiprocessmla.cpp │ ├── gstsimaaiprocessmla.h │ ├── gstsimaaiprocessmlastandalone.cpp │ ├── README.md │ └── res │ └── quantized_resnet50_stage1_mla.lm └── simaaisrc ├── CMakeLists.txt ├── gstsimaaisrc.cc ├── gstsimaaisrc.h └── README.md #. Update the ``.project/pluginsInfo.json`` with the path to the new plugin along with a new generated ``uuidgen`` id. * SKIP. We have not added a new plugin, so need to update the ``.project/pluginsInfo.json`` file. #. Update the ``application.json`` with the new plugin's ``name``, ``pluginGid`` (same as previous step), and ``[resources][configs]``: .. code-block:: json :caption: application.json :linenos: { "name": "resnet50_sample_name", "appId": "resnet50_sample_id", "sequence": 1, "pipelines": [ { "name": "resnet50_sample_pipeline_name", "sequence": 1, "plugins": [ { "name": "simaaisrc", "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "resources": { "configs": [] } }, { "name": "processcvu", "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a", "resources": { "configs": [ "cfg/genpreproc_200_cvu_cfg_params.json", "cfg/detessdequant_201_cvu_cfg_params.json" ] } }, { "name": "processmla", "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d", "resources": { "configs": [ "cfg/simaaiprocessmla_cfg_params.json" ], "lms": [ "res/quantized_resnet50_stage1_mla.lm" ] }, "dependencies": { "value": [ "dispatcher-lite.zip" ] } } ], "connections": [], "resources": [ { "name": "genpreproc_200_cvu_cfg_app", "location": "resources/genpreproc_200_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "detessdequant_201_cvu_cfg_app", "location": "resources/detessdequant_201_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "run_cvu_cfgs_sh", "location": "resources/run_cvu_cfgs.sh", "destination": { "provider": "RPM", "type": "bin" } } ], "dependencies": [ { "name": "run_cvu_cfgs_sh", "location": "/data/simaai/applications/resnet50_sample_name/bin/", "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh" } ] } ] } .. note:: Let's review the changes to the ``application.json``: * Added ``["pipelines"]["plugins"]["processcvu"]["resources"]: cfg/detessdequant_201_cvu_cfg_params.json``: * Makes our CVU graph configuration file available to the ``simaaiprocesscvu`` plugin. * Added ``["pipelines"]["resources"]["name"]: "detessdequant_201_cvu_cfg_app"``: * This makes our postprocessing CVU configuration binary available to our application. Adding ``argmax_print`` custom plugin ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description *********** In this step we will be adding the custom plugin ``argmax_print`` that we developed and tested in: * :ref:`developing_gstreamer_app_gstreamer_argmaxprint` Steps ***** #. Copy the plugin from |plugin_base| to ``/home/docker/sima-cli/resnet50_app/plugins/``: .. note:: * For any custom plugin that uses the aggregator template, we also need to copy over: * ``/usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/templates/`` to the root of our application directory. * We then need to edit the ``target_include_directories`` in ``plugins/argmax_print/CMakeLists.txt`` from ``${CMAKE_CURRENT_SOURCE_DIR}/../templates`` to ``${CMAKE_CURRENT_SOURCE_DIR}/templates``. .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/ plugins/ sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/templates/ plugins/argmax_print/ sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -L 2 plugins/ plugins/ ├── argmax_print │ ├── argmax_print_cfg.json │ ├── CMakeLists.txt │ ├── payload.cpp │ └── templates ├── processcvu │ ├── cfg │ ├── CMakeLists.txt │ ├── gstsimaaiprocesscvu.cpp │ ├── gstsimaaiprocesscvu.h │ └── README.md ├── processmla │ ├── cfg │ ├── CMakeLists.txt │ ├── gstsimaaiprocessmlacommon.h │ ├── gstsimaaiprocessmla.cpp │ ├── gstsimaaiprocessmla.h │ ├── gstsimaaiprocessmlastandalone.cpp │ ├── README.md │ └── res └── simaaisrc ├── CMakeLists.txt ├── gstsimaaisrc.cc ├── gstsimaaisrc.h └── README.md #. Create a ``cfg/`` directory: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mkdir plugins/argmax_print/cfg sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -L 2 plugins/ plugins/ ├── argmax_print │ ├── argmax_print_cfg.json │ ├── cfg │ ├── CMakeLists.txt │ ├── payload.cpp │ └── templates ├── processcvu │ ├── cfg │ ├── CMakeLists.txt │ ├── gstsimaaiprocesscvu.cpp │ ├── gstsimaaiprocesscvu.h │ └── README.md ├── processmla │ ├── cfg │ ├── CMakeLists.txt │ ├── gstsimaaiprocessmlacommon.h │ ├── gstsimaaiprocessmla.cpp │ ├── gstsimaaiprocessmla.h │ ├── gstsimaaiprocessmlastandalone.cpp │ ├── README.md │ └── res └── simaaisrc ├── CMakeLists.txt ├── gstsimaaisrc.cc ├── gstsimaaisrc.h └── README.md .. note:: The file ``argmax_print_cfg.json`` is the configuration file created when we developed the plugin. If you created it elsewhere, please locate it as we will need to copy it into the ``cfg`` directory. #. Create the configuration file inside the ``cfg/`` directory: During the development of the ``argmax_print`` custom plugin, a configuration file was written. That file was brought in when we copied the plugin (``argmax_print_cfg.json``). Move the configuration file into the ``cfg/`` directory: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mv plugins/argmax_print/argmax_print_cfg.json plugins/argmax_print/cfg sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -L 2 plugins/argmax_print plugins ├── argmax_print │ ├── cfg │ │ └── argmax_print_cfg.json │ ├── CMakeLists.txt │ ├── payload.cpp │ └── templates ├── processcvu │ ├── cfg │ │ ├── detessdequant_201_cvu_cfg_params.json │ │ └── genpreproc_200_cvu_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocesscvu.cpp │ ├── gstsimaaiprocesscvu.h │ └── README.md ├── processmla │ ├── cfg │ │ └── simaaiprocessmla_cfg_params.json │ ├── CMakeLists.txt │ ├── gstsimaaiprocessmlacommon.h │ ├── gstsimaaiprocessmla.cpp │ ├── gstsimaaiprocessmla.h │ ├── gstsimaaiprocessmlastandalone.cpp │ ├── README.md │ └── res │ └── quantized_resnet50_stage1_mla.lm └── simaaisrc ├── CMakeLists.txt ├── gstsimaaisrc.cc ├── gstsimaaisrc.h └── README.md #. Update the ``.project/pluginsInfo.json`` with the path to the new plugin along with a new generated ``uuidgen`` id. * Generate a new ``uuid`` for this plugin: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen b0e0910f-a58b-419f-a58d-584d94691836 * Update the ``.project/pluginsInfo.json`` to: .. code-block:: json :linenos: { "pluginsInfo" : [ { "gid" : "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "path" : "plugins/simaaisrc" }, { "gid" : "09b09793-e1de-4881-9fd8-7674b948864a", "path" : "plugins/processcvu" }, { "gid" : "c086febf-7f33-447d-a4ed-96be718de36d", "path" : "plugins/processmla" }, { "gid" : "b0e0910f-a58b-419f-a58d-584d94691836", "path" : "plugins/argmax_print" } ] } #. Update the ``application.json`` with the new plugin's ``name``, ``pluginGid`` (same as previous step), and ``[resources][configs]``: .. code-block:: json :caption: application.json :linenos: { "name": "resnet50_sample_name", "appId": "resnet50_sample_id", "sequence": 1, "pipelines": [ { "name": "resnet50_sample_pipeline_name", "sequence": 1, "plugins": [ { "name": "simaaisrc", "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "resources": { "configs": [] } }, { "name": "processcvu", "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a", "resources": { "configs": [ "cfg/genpreproc_200_cvu_cfg_params.json", "cfg/detessdequant_201_cvu_cfg_params.json" ] } }, { "name": "processmla", "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d", "resources": { "configs": [ "cfg/simaaiprocessmla_cfg_params.json" ], "lms": [ "res/quantized_resnet50_stage1_mla.lm" ] }, "dependencies": { "value": [ "dispatcher-lite.zip" ] } }, { "name": "argmax_print", "pluginGid": "b0e0910f-a58b-419f-a58d-584d94691836", "resources": { "configs": [ "cfg/argmax_print_cfg.json" ] } } ], "connections": [], "resources": [ { "name": "genpreproc_200_cvu_cfg_app", "location": "resources/genpreproc_200_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "detessdequant_201_cvu_cfg_app", "location": "resources/detessdequant_201_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "run_cvu_cfgs_sh", "location": "resources/run_cvu_cfgs.sh", "destination": { "provider": "RPM", "type": "bin" } } ], "dependencies": [ { "name": "run_cvu_cfgs_sh", "location": "/data/simaai/applications/resnet50_sample_name/bin/", "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh" } ] } ] } .. note:: Let's review the changes to the ``application.json``: * Added ``["pipelines"]["plugins"]["name"]: "argmax_print"`` with its corresponding uuid and configuration file. Adding the GStreamer Command ---------------------------- Now that we have defined our application plugins, resources and configurations, it is time to provide the runtime string that will run when the application is deployed. Similarly like we did when using the standard gstreamer ``gst-launch-1.0`` command, the gst-launch command will look like: .. code-block:: bash gst-launch-1.0 --gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib' -v \ simaaisrc mem-target=1 node-name=\"my_image_src\" location=\"/data/simaai/applications/resnet50_sample_name/etc/golden_retriever_207_rgb.bin\" num-buffers=1 ! \ simaaiprocesscvu source-node-name=\"my_image_src\" buffers-list=\"my_image_src\" config=\"/data/simaai/applications/resnet50_sample_name/etc/genpreproc_200_cvu_cfg_params.json\" ! \ simaaiprocessmla config=\"/data/simaai/applications/resnet50_sample_name/etc/simaaiprocessmla_cfg_params.json\" ! \ simaaiprocesscvu source-node-name=\"mla-resnet\" buffers-list=\"mla-resnet\" config=\"/data/simaai/applications/resnet50_sample_name/etc/detessdequant_201_cvu_cfg_params.json\" name=\"detessdequant\" ! \ argmax_print config=\"/data/simaai/applications/resnet50_sample_name/etc/argmax_print_cfg.json\" ! \ fakesink" .. note:: In reading the gst-launch string above, there is one notable difference from the one used in :ref:`developing_gstreamer_app`: * The paths are all different. They will depend on where our application is installed and where we have placed our resources relative to that path. Lastly, in JSON we cannot add a multi-line string, so in the ``application.json`` below the command will be written as a single line. It has been split above into multiple lines for readability. .. code-block:: json :caption: application.json :linenos: { "name": "resnet50_sample_name", "appId": "resnet50_sample_id", "sequence": 1, "pipelines": [ { "name": "resnet50_sample_pipeline_name", "sequence": 1, "plugins": [ { "name": "simaaisrc", "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "resources": { "configs": [] } }, { "name": "processcvu", "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a", "resources": { "configs": [ "cfg/genpreproc_200_cvu_cfg_params.json", "cfg/detessdequant_201_cvu_cfg_params.json" ] } }, { "name": "processmla", "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d", "resources": { "configs": [ "cfg/simaaiprocessmla_cfg_params.json" ], "lms": [ "res/quantized_resnet50_stage1_mla.lm" ] }, "dependencies": { "value": [ "dispatcher-lite.zip" ] } }, { "name": "argmax_print", "pluginGid": "b0e0910f-a58b-419f-a58d-584d94691836", "resources": { "configs": [ "cfg/argmax_print_cfg.json" ] } } ], "connections": [], "resources": [ { "name": "genpreproc_200_cvu_cfg_app", "location": "resources/genpreproc_200_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "detessdequant_201_cvu_cfg_app", "location": "resources/detessdequant_201_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "run_cvu_cfgs_sh", "location": "resources/run_cvu_cfgs.sh", "destination": { "provider": "RPM", "type": "bin" } } ], "dependencies": [ { "name": "run_cvu_cfgs_sh", "location": "/data/simaai/applications/resnet50_sample_name/bin/", "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh" } ], "gst": "gst-launch-1.0 --gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib' -v simaaisrc mem-target=1 node-name=\"my_image_src\" location=\"/data/simaai/applications/resnet50_sample_name/etc/golden_retriever_207_rgb.bin\" num-buffers=1 ! simaaiprocesscvu source-node-name=\"my_image_src\" buffers-list=\"my_image_src\" config=\"/data/simaai/applications/resnet50_sample_name/etc/genpreproc_200_cvu_cfg_params.json\" ! simaaiprocessmla config=\"/data/simaai/applications/resnet50_sample_name/etc/simaaiprocessmla_cfg_params.json\" ! simaaiprocesscvu source-node-name=\"mla-resnet\" buffers-list=\"mla-resnet\" config=\"/data/simaai/applications/resnet50_sample_name/etc/detessdequant_201_cvu_cfg_params.json\" name=\"detessdequant\" ! argmax_print config=\"/data/simaai/applications/resnet50_sample_name/etc/argmax_print_cfg.json\" ! fakesink" } ] } Adding Final Configurations and Resources ----------------------------------------- In this section, we will add two additional items to our ``application.json``:abbr: #. The input to the pipeline that we have been using during testing: ``golden_retriever_207_rgb.bin``. #. Configuration information required by the ``mpk`` build system. First, copy the ``golden_retriever_207_rgb.bin`` to the application resources directory. Directory should look like: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -L 2 . ├── application.json ├── core │ ├── allocator │ ├── CMakeLists.txt │ ├── dispatcher-lite │ └── simamm ├── plugins │ ├── argmax_print │ ├── processcvu │ ├── processmla │ └── simaaisrc ├── project.mpk └── resources ├── detessdequant_201_cvu_cfg_app ├── genpreproc_200_cvu_cfg_app ├── golden_retriever_207_rgb.bin └── run_cvu_cfgs.sh Second, update the ``application.json``: .. code-block:: json :caption: application.json :linenos: { "name": "resnet50_sample_name", "appId": "resnet50_sample_id", "sequence": 1, "pipelines": [ { "name": "resnet50_sample_pipeline_name", "sequence": 1, "plugins": [ { "name": "simaaisrc", "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3", "resources": { "configs": [] } }, { "name": "processcvu", "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a", "resources": { "configs": [ "cfg/genpreproc_200_cvu_cfg_params.json", "cfg/detessdequant_201_cvu_cfg_params.json" ] } }, { "name": "processmla", "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d", "resources": { "configs": [ "cfg/simaaiprocessmla_cfg_params.json" ], "lms": [ "res/quantized_resnet50_stage1_mla.lm" ] }, "dependencies": { "value": [ "dispatcher-lite.zip" ] } }, { "name": "argmax_print", "pluginGid": "b0e0910f-a58b-419f-a58d-584d94691836", "resources": { "configs": [ "cfg/argmax_print_cfg.json" ] } } ], "connections": [], "resources": [ { "name": "genpreproc_200_cvu_cfg_app", "location": "resources/genpreproc_200_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "detessdequant_201_cvu_cfg_app", "location": "resources/detessdequant_201_cvu_cfg_app", "destination": { "provider": "RPM", "type": "config" } }, { "name": "run_cvu_cfgs_sh", "location": "resources/run_cvu_cfgs.sh", "destination": { "provider": "RPM", "type": "bin" } }, { "name": "golden_retriever_207_rgb.bin", "location": "resources/golden_retriever_207_rgb.bin", "destination": { "provider": "RPM", "type": "config" } } ], "dependencies": [ { "name": "run_cvu_cfgs_sh", "location": "/data/simaai/applications/resnet50_sample_name/bin/", "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh" } ], "gst": "gst-launch-1.0 --gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib' -v simaaisrc mem-target=1 node-name=\"my_image_src\" location=\"/data/simaai/applications/resnet50_sample_name/etc/golden_retriever_207_rgb.bin\" num-buffers=1 ! simaaiprocesscvu source-node-name=\"my_image_src\" buffers-list=\"my_image_src\" config=\"/data/simaai/applications/resnet50_sample_name/etc/genpreproc_200_cvu_cfg_params.json\" ! simaaiprocessmla config=\"/data/simaai/applications/resnet50_sample_name/etc/simaaiprocessmla_cfg_params.json\" ! simaaiprocesscvu source-node-name=\"mla-resnet\" buffers-list=\"mla-resnet\" config=\"/data/simaai/applications/resnet50_sample_name/etc/detessdequant_201_cvu_cfg_params.json\" name=\"detessdequant\" ! argmax_print config=\"/data/simaai/applications/resnet50_sample_name/etc/argmax_print_cfg.json\" ! fakesink" } ], "configuration": { "installationPrefixes": { "configurations": "/data/simaai/applications/resnet50_sample_name/etc", "binaries": "/data/simaai/applications/resnet50_sample_name/bin", "resources": "/data/simaai/applications/resnet50_sample_name/share", "libraries": "/data/simaai/applications/resnet50_sample_name/lib", "globalLibs": "/usr/lib" }, "compilation": { "copyPokyLibs": false }, "gst": { "options": [ "--gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib'" ] }, "environment": [ "GST_DEBUG=0", "LD_LIBRARY_PATH='/data/simaai/applications/resnet50_sample_name/lib'" ] }, "manifestVersion": "2.0.0" } .. note:: Let's review the changes to the ``application.json``: * Added ``["configuration"]["installationPrefixes"]["configurations" | "binaries" | "resources" | "libraries" | "globalLibs"]`` * This is the main paths where we will store various files and resources for our application. Standard to leave as shown here but change the name of the application. * Added ``["configuration"]["compilation"]["copyPokyLibs"]: false`` * This is telling the system that the necessary libs are already on pre-built on the board and do not need to be copied. * Added ``["configuration"]["gst"]["options"]: "--gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib'"`` * This is the location of where our plugins will be copied to by the mpk build system. * Added ``["configuration"]["environment"]["GST_DEBUG" | "LD_LIBRARY_PATH"]:`` * These are any environment variables we want to set during our application launch. * Added ``["manifestVersion"]: 2.0.0`` * The version of the application.json manifest, and how it will be interpreted. For this version of the guide, always use this manifest version in all of your applications. Create the MPK -------------- Now that we have fully tested and defined our application, we are ready to package it into an ``mpk`` file. To do so we will use the command ``mpk create`` as referenced in :ref:`mpk tool`. Simply step one directory our of the application project directory and invoke the command: .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cd .. sima-user@docker-image-id:/home/docker/sima-cli$ mpk create -s resnet50_app/ -d resnet50_app/ --clean ℹ Cleaning up build artifacts... ✔ Successfully cleaned up build artifacts. ℹ Step a65-apps COMPILE completed successfully. ℹ Step COMPILE completed successfully. ℹ Step COPY RESOURCE completed successfully ℹ Step RPM BUILD completed successfully. ✔ Successfully created MPK at '/home/docker/sima-cli/resnet50_app/project.mpk' ℹ Cleaning up build artifacts... ✔ Successfully cleaned up build artifacts. sima-user@docker-image-id:/home/docker/sima-cli$ tree -L 1 resnet50_app/ resnet50_app/ ├── application.json ├── core ├── plugins ├── project.mpk └── resources The ``project.mpk`` is the packaged up application that is ready to be installed. You can rename this if you wish. Deploy the MPK -------------- Lastly, in order to deploy our application to our device, we will make use of the command ``mpk deploy`` as referenced in :ref:`mpk tool`. .. note:: In order to perform this step, make sure you have used ``mpk device connect`` and currently have a device successfully connected. .. code-block:: console sima-user@docker-image-id:/home/docker/sima-cli$ mpk device list Active Connections over Ethernet ┏━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓ ┃ Status ┃ Status Details ┃ Target ┃ Username ┃ Device ┃ Jump Server ┃ ┡━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩ │ 🟢 │ Online │ 10.0.0.143 │ sima │ fence_wonder │ - │ └────────┴────────────────┴────────────┴──────────┴──────────────┴─────────────┘ ❕ There are no PCIe devices available. sima-user@docker-image-id:/home/docker/sima-cli$ mpk deploy -d fence_wonder -f resnet50_app/project.mpk 🚀 Sending MPK to 10.0.0.143... Transfer Progress for project.mpk: 100.00% 🏁 MPK sent successfully! ✔ MPK Deployed! ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ✔ MPK Deployment is successful for project.mpk. At this point, we have deployed the ``mpk`` successfully on the board, and the application should have executed right away. We can verify that the application executed as expected by checking the logs. On the MLSoC device: #. Navigate to ``/tmp/resnet50_sample_pipeline_name`` (name of our pipeline in the application.json) #. Print the ``gst-launch`` output: .. code-block:: console davinci:/tmp/resnet50_sample_pipeline_name$ cat gst-launch-1.0 ... Index of largest value: 207 Largest value: 0.959188 Pipeline is PREROLLED ... Setting pipeline to PLAYING ... Redistribute latency... New clock: GstSystemClock Got EOS from element "pipeline0". Execution ended after 0:00:00.001224352 Setting pipeline to NULL ... Freeing pipeline ... At this point we have learned how to develop, test, debug, and package an application target to an MLSoC device. Future examples and application notes will contain more advanced applications, but all of the essential concepts have been covered in this guide.