Developing the MPK Application

An mpk application is used to conveniently package and deploy an application to MLSoC target devices using MPK CLI tools included with Palette software. An MPK Application is just a package that executes a GStreamer application at runtime.

This section describes how we can create an end to end working ML Pipeline Package (mpk) in a step-by-step manner. We will finish our ResNet50 application by packaging it up into an mpk and deploying it on the board.

To create an application mpk for the MLSoC:

  • Test and debug an application using gst-launch-1.0 - or start from an existing example application and edit.

  • Define a project structure along with necessary configuration files to define the application. The compile and package using mpk create CLI tool.

  • Deploy the application using mpk deploy CLI tool to deploy the created mpk application to the MLSoC board.

Important

The creation of an MPK Application happens within the Palette docker container. All plugins and resources are cross-compiled and packaged into a resulting .mpk package with the use of the mpk create command as we will see later in this guide.

Folder Structure

Creating an MPK Application requires a specific folder structure. Every folder serves a purpose, let’s discuss them in detail. The folder tree is as follows:

application_name
├── plugins
├── .project
│   └── pluginsInfo.json
├── resources
├── core
└── application.json

application_name

  • This is the master folder for the target application we are developing and holds all necessary resources inside.

plugins

  • This directory holds all the different GStreamer Plugins plugins being used in the pipeline.

  • Plugins needed for our application are located in /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base as we have seen during our GStreamer development and debugging.

.project

  • This is a hidden directory internally required by the mpk create command while doing a compilation of plugins available in the /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base.

  • This directory holds a very important file named pluginsInfo.json. This file contains all the plugins’ list which are being used in the MPK Application.

resources

  • This directory is keeping the CVU Configuration Application required to load different EV74 graphs. It also holds any necessary dependencies that need to be executed before actually running the pipeline on target device.

core

  • This directory holds the dispatcher present under core directory at /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base.

  • This dispatcher is necessary for dispatching data on MLA and is required by simaaiprocessmla.

application.json

  • This is a complete description of the application in JSON format. This file holds various information related to pipeline’s name, plugins, gstreamer command, etc.

Pre-Requisites

Before creating an MPK Application, we should:

  1. Have a reference of the application we are developing. In our case, this will be the Reference Classification Application.

  2. Have debugged the application using GStreamer gst-launch strings to ensure everything is working as expected.

  3. Should have the latest version of Palette installed with the corresponding firmware on the MLSoC.

Let’s remind ourselves of the application we will package using the an mpk:

The application logic:

../../../_images/resnet50_application_with_images.jpg

Mapping it to the MLSoC:

../../../_images/resnet50_application_hw.jpg

Step-by-Step Guide

Create the directory structure

As discussed above in the previous section, let’s create the folder structure for our ResNet50 Sample App.

  1. Create project directory structure

    sima-user@docker-image-id:~$ cd /home/docker/sima-cli
sima-user@docker-image-id:/home/docker/sima-cli$ mkdir resnet50_app && cd resnet50_app
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mkdir plugins .project resources && touch application.json .project/pluginsInfo.json
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3
.
├── application.json
├── plugins
├── .project
│   └── pluginsInfo.json
└── resources

  2. Add some boilerplate in .project/pluginsInfo.json running the following command under /home/docker/sima-cli/resnet50_app:

    echo '{
    "pluginsInfo": []
}' > .project/pluginsInfo.json

  3. Add some boilerplate in application.json running the following command under /home/docker/sima-cli/resnet50_app:

    echo '{
    "name": "resnet50_sample_name",
    "appId": "resnet50_sample_id",
    "sequence": 1,
    "pipelines": [
        {
            "name": "resnet50_sample_pipeline_name",
            "sequence": 1,
            "plugins": [],
            "connections": [],
            "resources": [],
            "dependencies": []
        }
    ]
}' > application.json

    Note

    Some notable parameters:

    • ["name"]: Name of the application as it will be installed on the MLSoC device.

    • ["appID"]: Unique ID for the application to differentiate from other applications.

    • ["pipelines"]: Where the pipeline definition will go. Only support one pipeline at the moment.

    • ["resources"]: Any resources that the application depends on (for example: CVU configuration applications).

    • ["dependencies"]: Any programs or commands that the application has to run at runtime.

Install uuidgen

  1. To create GUID for every plugin, one needs to use uuidgen tool which needs to be installed additionally if not present.

    sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ sudo apt-get update && sudo apt-get install uuid-runtime

  2. For every plugin specification, generate the GUID as shown below:

    sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen
1627c73c-bf68-49af-b9d4-89fb6c13dad5

Adding Plugins

How to add a new plugin

Now that we have the skeleton structure of creating an MPK Application, let’s begin adding the plugins of the pipeline.

Overall, the process of adding each plugin is as follows:

  1. Copy the plugin from /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base to our plugins directory /home/docker/sima-cli/resnet50_app/plugins/

    • Bring the sources of the plugin into the application for the mpk create framework to cross-compile the plugin.

    • If using the same plugin more than once, no need to copy it multiple times.

  2. Create a cfg/ directory for your plugin configurations inside of the copied plugin folder (not always necessary)

    • This is where we will store any JSONs or configuration files associated with this plugin

    • For the simaaiprocesscvu to run CVU Graphs on the EV74 CVU, this is also where we will set up the configuration application

  3. Create the configuration file inside the cfg/ directory (not always necessary)

    • Location to store configuration files for the plugin

  4. Update the .project/pluginsInfo.json with the path to the new plugin along with a new generated uuidgen id

    • This creates a mapping of our plugin directory to a unique ID recognizable by the mpk create framework

  5. Update the application.json with the new plugin’s name, pluginGid (same as previous step), and [resources][configs]

    • This step associates a created plugin and it’s configuration to the overall application.

Plugins for our ResNet50 pipeline

If we recall from Mapping application to the MLSoC, we will need the following plugins as part of our application:

simaaisrc -> simaaiprocesscvu -> simaaiprocessmla -> simaaiprocesscvu -> argmax_print -> fakesink

Where:

  • argmax_print is our custom plugin

  • fakesink is an open source GStreamer plugin that we do not have to manually compile or copy over to our project structure.

Adding SiMa GStreamer Plugins

Adding simaaisrc

Description

In this step we will be adding the file source to our pipeline as it was tested in:

Steps

  1. Copy the plugin from /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base to /home/docker/sima-cli/resnet50_app/plugins/:

    sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/simaaisrc plugins/
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3
.
├── application.json
├── plugins
│   └── simaaisrc
│       ├── CMakeLists.txt
│       ├── gstsimaaisrc.cc
│       ├── gstsimaaisrc.h
│       └── README.md
├── .project
│   └── pluginsInfo.json
└── resources

    Note

    If you have previously compiled the plugin for testing, you will also see a build/ directory in the plugins/simaaisrc/ directory.

  2. Create a cfg/ directory:

    • SKIP. This plugin does not require any configuration files as all configurations will be done via command line parameters later.

  3. Create the configuration file inside the cfg/ directory:

    • SKIP. This plugin does not require any configuration files as all configurations will be done via command line parameters later.

  4. Update the .project/pluginsInfo.json with the path to the new plugin along with a new generated uuidgen id.

    • Generate a new uuid for this plugin:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen
0c255b9a-e5a8-4346-993f-e3527cc6d4e3

    • Update the .project/pluginsInfo.json to:

      1{
2    "pluginsInfo" : [
3        {
4            "gid" : "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
5            "path" : "plugins/simaaisrc"
6        }
7    ]
8}

  5. Update the application.json with the new plugin’s name, pluginGid (same as previous step), and [resources][configs]:

    application.json
     1{
 2    "name": "resnet50_sample_name",
 3    "appId": "resnet50_sample_id",
 4    "sequence": 1,
 5    "pipelines": [
 6        {
 7            "name": "resnet50_sample_pipeline_name",
 8            "sequence": 1,
 9            "plugins": [
10                {
11                    "name": "simaaisrc",
12                    "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
13                    "resources": {
14                        "configs": []
15                    }
16                }
17            ],
18            "connections": [],
19            "resources": [],
20            "dependencies": []
21        }
22    ]
23}

Adding simaaiprocesscvu (preprocessing)

Description

In this step we will be adding the ResNet50 preprocessing to our pipeline as it was tested in:

Steps

  1. Copy the plugin from /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base to /home/docker/sima-cli/resnet50_app/plugins/:

    sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/processcvu/ plugins/
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3
.
├── application.json
├── plugins
│   ├── processcvu
│   │   ├── CMakeLists.txt
│   │   ├── gstsimaaiprocesscvu.cpp
│   │   ├── gstsimaaiprocesscvu.h
│   │   └── README.md
│   └── simaaisrc
│       ├── CMakeLists.txt
│       ├── gstsimaaisrc.cc
│       ├── gstsimaaisrc.h
│       └── README.md
├── .project
│   └── pluginsInfo.json
└── resources

  2. Create a cfg/ directory:

    sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mkdir plugins/processcvu/cfg
sima-user@docker-image-ide:/home/docker/sima-cli/resnet50_app$ tree -aL 3
.
├── application.json
├── plugins
│   ├── processcvu
│   │   ├── cfg
│   │   ├── CMakeLists.txt
│   │   ├── gstsimaaiprocesscvu.cpp
│   │   ├── gstsimaaiprocesscvu.h
│   │   └── README.md
│   └── simaaisrc
│       ├── CMakeLists.txt
│       ├── gstsimaaisrc.cc
│       ├── gstsimaaisrc.h
│       └── README.md
├── .project
│   └── pluginsInfo.json
└── resources

  3. Create the configuration file inside the cfg/ directory:

    • Copy the JSON genpreproc_200_cvu_cfg_params.json configuration file to this location or re-create the file.

    • To copy it:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp /home/docker/sima-cli/ev74_cgfs/sima_generic_preproc/genpreproc_200_cvu_cfg_params.json plugins/processcvu/cfg/

      Note

      Copying the configuration file depends on having followed the pre-requisite guide: Step 1: Run and verify output of simaaiprocesscvu (CVU preprocess) found under Developing & Debugging GStreamer Applications

    • To recreate it run the following command under /home/docker/sima-cli/resnet50_app:

      echo '{
    "version": 0.1,
    "node_name": "generic_preproc",
    "simaai__params": {
        "params": 15,
        "cpu": 1,
        "next_cpu": 4,
        "no_of_outbuf": 1,
        "ibufname": "null",
        "graph_id": 200,
        "img_width": 1920,
        "img_height": 1080,
        "input_width": 1920,
        "input_height": 1080,
        "output_width": 224,
        "output_height": 224,
        "scaled_width": 224,
        "scaled_height": 224,
        "batch_size": 1,
        "normalize": 1,
        "rgb_interleaved": 1,
        "aspect_ratio": 0,
        "tile_width": 32,
        "tile_height": 16,
        "input_depth": 3,
        "output_depth": 3,
        "quant_zp": -14,
        "quant_scale": 53.59502566491281,
        "mean_r": 0.406,
        "mean_g": 0.456,
        "mean_b": 0.485,
        "std_dev_r": 0.225,
        "std_dev_g": 0.224,
        "std_dev_b": 0.229,
        "input_type": 2,
        "output_type": 0,
        "scaling_type": 3,
        "offset": 150528,
        "padding_type": 4,
        "input_stride": 0,
        "output_stride": 0,
        "output_dtype": 0,
        "debug": 0,
        "out_sz": 301056,
        "dump_data": 1
    }
}' > plugins/processcvu/cfg/genpreproc_200_cvu_cfg_params.json

      Warning

      You should change the dump_data parameter to 0 when deploying the final application. We have it set to 1 currently to ensure we can test when we deploy that it works as expected.

    • The new directory structure should look like this:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 4
.
├── application.json
├── plugins
│   ├── processcvu
│   │   ├── cfg
│   │   │   └── genpreproc_200_cvu_cfg_params.json
│   │   ├── CMakeLists.txt
│   │   ├── gstsimaaiprocesscvu.cpp
│   │   ├── gstsimaaiprocesscvu.h
│   │   └── README.md
│   └── simaaisrc
│       ├── CMakeLists.txt
│       ├── gstsimaaisrc.cc
│       ├── gstsimaaisrc.h
│       └── README.md
├── .project
│   └── pluginsInfo.json
└── resources

    • Add the CVU configuration application for this plugin from: Step 1: Run and verify output of simaaiprocesscvu (CVU preprocess). We will first copy the compiled binary for the application and then we will add a small shell script to run the app at runtime.

      • Copy the pre-compiled application to the resources folder:

        sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp /home/docker/sima-cli/ev74_cgfs/sima_generic_preproc/build/genpreproc_200_cvu_cfg_app resources/

        Note

        Copying the configuration app depends on having followed the pre-requisite guide: Step 1: Run and verify output of simaaiprocesscvu (CVU preprocess) found under Developing & Debugging GStreamer Applications

      • Create a shell script:

        This shell script will run before the GStreamer pipeline kicks off. To generate the file run the following command under /home/docker/sima-cli/resnet50_app:

        echo '
#!/bin/sh

APP_DIR="/data/simaai/applications"
APP_NAME="resnet50_sample_name"
ETC_PATH="${APP_DIR}/${APP_NAME}/etc"
PREPROC_CVU_CONFIG_BIN="genpreproc_200_cvu_cfg_app"
PREPROC_CVU_CONFIG_JSON="genpreproc_200_cvu_cfg_params.json"

# Execute preprocessing config
${ETC_PATH}/${PREPROC_CVU_CONFIG_BIN} ${ETC_PATH}/${PREPROC_CVU_CONFIG_JSON}
' > resources/run_cvu_cfgs.sh

        Note

        This script is similar to the ones we created in the Step 1: Run and verify output of simaaiprocesscvu (CVU preprocess) guide, with two notable differences:

        • The directories are specific to where an** mpk application is installed.

          • All mpk applications are installed under /data/simaai/applications/. This application will be installed under /data/simaai/applications/resnet50_sample_name/.

        • The script purely invokes the CVU configuration application, and does not call gst-launch-1.0.

      • The new directory structure should look like this:

        sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 4
.
├── application.json
├── plugins
│   ├── processcvu
│   │   ├── cfg
│   │   │   └── genpreproc_200_cvu_cfg_params.json
│   │   ├── CMakeLists.txt
│   │   ├── gstsimaaiprocesscvu.cpp
│   │   ├── gstsimaaiprocesscvu.h
│   │   └── README.md
│   └── simaaisrc
│       ├── CMakeLists.txt
│       ├── gstsimaaisrc.cc
│       ├── gstsimaaisrc.h
│       └── README.md
├── .project
│   └── pluginsInfo.json
└── resources
    ├── genpreproc_200_cvu_cfg_app
    └── run_cvu_cfgs.sh

  4. Update the .project/pluginsInfo.json with the path to the new plugin along with a new generated uuidgen id.

    • Generate a new uuid for this plugin:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen
09b09793-e1de-4881-9fd8-7674b948864a

    • Update the .project/pluginsInfo.json to:

       1{
 2    "pluginsInfo" : [
 3        {
 4            "gid" : "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
 5            "path" : "plugins/simaaisrc"
 6        },
 7        {
 8            "gid" : "09b09793-e1de-4881-9fd8-7674b948864a",
 9            "path" : "plugins/processcvu"
10        }
11    ]
12}

  5. Update the application.json with the new plugin’s name, pluginGid (same as previous step), and [resources][configs]:

    application.json
     1{
 2    "name": "resnet50_sample_name",
 3    "appId": "resnet50_sample_id",
 4    "sequence": 1,
 5    "pipelines": [
 6        {
 7            "name": "resnet50_sample_pipeline_name",
 8            "sequence": 1,
 9            "plugins": [
10                {
11                    "name": "simaaisrc",
12                    "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
13                    "resources": {
14                        "configs": []
15                    }
16                },
17                {
18                    "name": "processcvu",
19                    "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a",
20                    "resources": {
21                        "configs": [
22                            "cfg/genpreproc_200_cvu_cfg_params.json"
23                        ]
24                    }
25                }
26            ],
27            "connections": [],
28            "resources": [
29                {
30                    "name": "genpreproc_200_cvu_cfg_app",
31                    "location": "resources/genpreproc_200_cvu_cfg_app",
32                    "destination": {
33                        "provider": "RPM",
34                        "type": "config"
35                    }
36                },
37                {
38                    "name": "run_cvu_cfgs_sh",
39                    "location": "resources/run_cvu_cfgs.sh",
40                    "destination": {
41                        "provider": "RPM",
42                        "type": "bin"
43                    }
44                }
45            ],
46            "dependencies": [
47                {
48                    "name": "run_cvu_cfgs_sh",
49                    "location": "/data/simaai/applications/resnet50_sample_name/bin/",
50                    "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh"
51                }
52            ]
53        }
54    ]
55}

    Note

    We have made a large number of updates to the application.json so let us review the changes:

    • Added ["pipelines"]["plugins"]["name"]: "processcvu":

      • Just like we did before, we are making our application aware of our new processcvu plugin we want to use.

    • Added ["pipelines"]["plugins"]["resources"]["configs"]: "cfg/genpreproc_200_cvu_cfg_params.json"

      • Here we are providing the JSON configuration file that is needed to correctly initialize the plugin with the correct parameters.

    • Added ["pipelines"]["resources"]["name"]: "genpreproc_200_cvu_cfg_app":

      • This makes our preprocessing CVU configuration binary available to our application.

    • Added ["pipelines"]["resources"]["name"]: "run_cvu_cfgs_sh":

      • This makes our preprocessing CVU configuration shell script available to our application.

    • Added ["pipelines"]["dependencies"]["name"]: "run_cvu_cfgs_sh":

      • This ensures that the application will run the run_cvu_cfgs_sh before executing the GStreamer application as specified by the ["pipelines"]["dependencies"]["command"] field under ["pipelines"]["dependencies"][0] (run_cvu_cfgs_sh).

Adding simaaiprocessmla

Description

In this step we will be adding the ResNet50 inferencing to our pipeline as it was tested in:

Steps

  1. Copy the plugin from /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base to /home/docker/sima-cli/resnet50_app/plugins/:

    Note

    • For the simaaiprocessmla plugin, we also need to copy over:

      • /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/core to the root of our application directory as well.

    sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/processmla/ plugins/
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/core .
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$  tree -aL 2
.
├── application.json
├── core
│   ├── allocator
│   ├── CMakeLists.txt
│   ├── dispatcher-lite
│   └── simamm
├── plugins
│   ├── processcvu
│   ├── processmla
│   └── simaaisrc
├── .project
│   └── pluginsInfo.json
└── resources
    ├── genpreproc_200_cvu_cfg_app
    └── run_cvu_cfgs.sh

  2. Create a cfg/ and res/ directory:

    Note

    • For the simaaiprocessmla plugin, we also need to create a res/ directory:

      • We need a resource specifically for the plugin (the compiled model lm), so we will create a resource directory res inside of our plugin as well.

      • Why not place lm model inside the /home/docker/sima-cli/resnet50_app/resources?

        • Because that is reserved for application level resources, not resources needed for a specific plugin.

    • Create the directories:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mkdir plugins/processmla/cfg plugins/processmla/res
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 plugins/
plugins/
├── processcvu
│   ├── cfg
│   │   └── genpreproc_200_cvu_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocesscvu.cpp
│   ├── gstsimaaiprocesscvu.h
│   └── README.md
├── processmla
│   ├── cfg
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocessmlacommon.h
│   ├── gstsimaaiprocessmla.cpp
│   ├── gstsimaaiprocessmla.h
│   ├── gstsimaaiprocessmlastandalone.cpp
│   ├── README.md
│   └── res
└── simaaisrc
    ├── CMakeLists.txt
    ├── gstsimaaisrc.cc
    ├── gstsimaaisrc.h
    └── README.md

    • Copy the lm model to the new res directory:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp <path to output of ModelSDK>/compiled_resnet50/quantized_resnet50_mpk/quantized_resnet50_stage1_mla.lm plugins/processmla/res/
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 plugins/
plugins/
├── processcvu
│   ├── cfg
│   │   └── genpreproc_200_cvu_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocesscvu.cpp
│   ├── gstsimaaiprocesscvu.h
│   └── README.md
├── processmla
│   ├── cfg
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocessmlacommon.h
│   ├── gstsimaaiprocessmla.cpp
│   ├── gstsimaaiprocessmla.h
│   ├── gstsimaaiprocessmlastandalone.cpp
│   ├── README.md
│   └── res
│       └── quantized_resnet50_stage1_mla.lm
└── simaaisrc
    ├── CMakeLists.txt
    ├── gstsimaaisrc.cc
    ├── gstsimaaisrc.h
    └── README.md

      Note

      The lm file is retrieved by extracting the contents of the tar.gz output of the ModelSDK.

  3. Create the configuration file inside the cfg/ directory:

    • To recreate the simaaiprocessmla configuration file, run the following command under /home/docker/sima-cli/resnet50_app:

      echo '{
    "version" : 0.1,
    "node_name" : "mla-resnet",
    "simaai__params" : {
        "params" : 15,
        "index" : 1,
        "cpu" : 4,
        "next_cpu" : 1,
        "out_sz" : 1008,
        "no_of_outbuf" : 1,
        "batch_size" : 1,
        "batch_sz_model" : 1,
        "in_tensor_sz": 0,
        "out_tensor_sz": 0,
        "ibufname" : "generic_preproc",
        "model_path" : "/home/sima/resnet50_example_app/models/quantized_resnet50_stage1_mla.lm",
        "debug" : 0,
        "dump_data": 1
    }
}' > plugins/processmla/cfg/simaaiprocessmla_cfg_params.json

      Warning

      You should change the dump_data parameter to 0 when deploying the final application. We have it set to 1 currently to ensure we can test when we deploy that it works as expected.

    • The directory structure should now look like this:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 plugins/
plugins/
├── processcvu
│   ├── cfg
│   │   └── genpreproc_200_cvu_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocesscvu.cpp
│   ├── gstsimaaiprocesscvu.h
│   └── README.md
├── processmla
│   ├── cfg
│   │   └── simaaiprocessmla_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocessmlacommon.h
│   ├── gstsimaaiprocessmla.cpp
│   ├── gstsimaaiprocessmla.h
│   ├── gstsimaaiprocessmlastandalone.cpp
│   ├── README.md
│   └── res
│       └── quantized_resnet50_stage1_mla.lm
└── simaaisrc
    ├── CMakeLists.txt
    ├── gstsimaaisrc.cc
    ├── gstsimaaisrc.h
    └── README.md

  4. Update the .project/pluginsInfo.json with the path to the new plugin along with a new generated uuidgen id.

    • Generate a new uuid for this plugin:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen
c086febf-7f33-447d-a4ed-96be718de36d

    • Update the .project/pluginsInfo.json to:

       1{
 2    "pluginsInfo" : [
 3        {
 4            "gid" : "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
 5            "path" : "plugins/simaaisrc"
 6        },
 7        {
 8            "gid" : "09b09793-e1de-4881-9fd8-7674b948864a",
 9            "path" : "plugins/processcvu"
10        },
11        {
12            "gid" : "c086febf-7f33-447d-a4ed-96be718de36d",
13            "path" : "plugins/processmla"
14        }
15    ]
16}

  5. Update the application.json with the new plugin’s name, pluginGid (same as previous step), and [resources][configs]:

    application.json
     1{
 2    "name": "resnet50_sample_name",
 3    "appId": "resnet50_sample_id",
 4    "sequence": 1,
 5    "pipelines": [
 6        {
 7            "name": "resnet50_sample_pipeline_name",
 8            "sequence": 1,
 9            "plugins": [
10                {
11                    "name": "simaaisrc",
12                    "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
13                    "resources": {
14                        "configs": []
15                    }
16                },
17                {
18                    "name": "processcvu",
19                    "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a",
20                    "resources": {
21                        "configs": [
22                            "cfg/genpreproc_200_cvu_cfg_params.json"
23                        ]
24                    }
25                },
26                {
27                    "name": "processmla",
28                    "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d",
29                    "resources": {
30                        "configs": [
31                            "cfg/simaaiprocessmla_cfg_params.json"
32                        ],
33                        "lms": [
34                            "res/quantized_resnet50_stage1_mla.lm"
35                        ]
36                    },
37                    "dependencies": {
38                        "value": [
39                            "dispatcher-lite.zip"
40                        ]
41                    }
42                }
43            ],
44            "connections": [],
45            "resources": [
46                {
47                    "name": "genpreproc_200_cvu_cfg_app",
48                    "location": "resources/genpreproc_200_cvu_cfg_app",
49                    "destination": {
50                        "provider": "RPM",
51                        "type": "config"
52                    }
53                },
54                {
55                    "name": "run_cvu_cfgs_sh",
56                    "location": "resources/run_cvu_cfgs.sh",
57                    "destination": {
58                        "provider": "RPM",
59                        "type": "bin"
60                    }
61                }
62            ],
63            "dependencies": [
64                {
65                    "name": "run_cvu_cfgs_sh",
66                    "location": "/data/simaai/applications/resnet50_sample_name/bin/",
67                    "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh"
68                }
69            ]
70        }
71    ]
72}

    Note

    Let’s review the changes to the application.json:

    • Added ["pipelines"]["plugins"]["name"]: "processmla":

      • Just like we did before, we are making our application aware of our new processmla plugin we want to use.

    • Added ["pipelines"]["plugins"]["processmla"]["resources"]["configs"]:

      • Here we are providing the JSON configuration file that is needed to correctly initialize the plugin with the correct parameters.

    • Added ["pipelines"]["plugins"]["processmla"]["resources"]["lms"]:

      • The processmla plugin depends on a compiled lm model to run. Here we specify the name and location of that model.

    • Added ["pipelines"]["plugins"]["processmla"]["dependencies"]:

      • The processmla plugin depends the dispatcher library. mpk create will create a zip and place it in the right location for us.

Adding simaaiprocesscvu (postprocessing)

Description

In this step we will be adding the ResNet50 preprocessing to our pipeline as it was tested in:

Steps

  1. Copy the plugin from /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base to /home/docker/sima-cli/resnet50_app/plugins/:

    • SKIP. We already copied the simaaiprocesscvu plugin earlier to our project, no need to do it again.

  2. Create a cfg/ directory:

    • SKIP. We already created a cfg/ directory inside the processcvu directory, no need to do it again.

  3. Create the configuration file inside the cfg/ directory:

    • Copy the JSON detessdequant_201_cvu_cfg_params.json configuration file to this location or re-create the file.

    • To copy it:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ ccp /home/docker/sima-cli/ev74_cgfs/sima_detess_dequant/detessdequant_201_cvu_cfg_params.json plugins/processcvu/cfg/

      Note

      Copying the configuration file depends on having followed the pre-requisite guide: Step 3: Run and verify output of simaaiprocesscvu (CVU postprocess) found under Developing & Debugging GStreamer Applications

    • To recreate it, run the following command under /home/docker/sima-cli/resnet50_app:

      echo '{
    "version": 0.1,
    "node_name": "detess-dequant",
    "simaai__params": {
        "params": 15,
        "cpu": 1,
        "next_cpu": 0,
        "no_of_outbuf": 1,
        "ibufname": "",
        "graph_id": 201,
        "img_width": 1280,
        "img_height": 720,
        "num_tensors": 1,
        "input_width": [1],
        "input_height": [1],
        "input_depth": [1000],
        "slice_width": [1],
        "slice_height": [1],
        "slice_depth": [1000],
        "dq_scale": [255.02200010497842],
        "dq_zp": [-128],
        "data_type": [0],
        "fp16_out_en": [0],
        "output_format": [0],
        "debug": 0,
        "out_sz": 4000,
        "dump_data": 1
    }
}' > plugins/processcvu/cfg/detessdequant_201_cvu_cfg_params.json

      Warning

      You should change the dump_data parameter to 0 when deploying the final application. We have it set to 1 currently to ensure we can test when we deploy that it works as expected.

    • The new directory structure should look like this:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 3 plugins/
plugins/
├── processcvu
│   ├── cfg
│   │   ├── detessdequant_201_cvu_cfg_params.json
│   │   └── genpreproc_200_cvu_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocesscvu.cpp
│   ├── gstsimaaiprocesscvu.h
│   └── README.md
├── processmla
│   ├── cfg
│   │   └── simaaiprocessmla_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocessmlacommon.h
│   ├── gstsimaaiprocessmla.cpp
│   ├── gstsimaaiprocessmla.h
│   ├── gstsimaaiprocessmlastandalone.cpp
│   ├── README.md
│   └── res
│       └── quantized_resnet50_stage1_mla.lm
└── simaaisrc
    ├── CMakeLists.txt
    ├── gstsimaaisrc.cc
    ├── gstsimaaisrc.h
    └── README.md

    • Add the CVU configuration application for this plugin from: Step 3: Run and verify output of simaaiprocesscvu (CVU postprocess). We will first copy the compiled binary for the application and then we will add a small shell script to run the app at runtime.

      • Copy the pre-compiled application to the resources folder:

        sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp /home/docker/sima-cli/ev74_cgfs/sima_detess_dequant/build/detessdequant_201_cvu_cfg_app resources/

        Note

        Copying the configuration app depends on having followed the pre-requisite guide: Step 3: Run and verify output of simaaiprocesscvu (CVU postprocess) found under Developing & Debugging GStreamer Applications

      • Run the following command under /home/docker/sima-cli/resnet50_app to modify the shell script to add how to run the postprocessing CVU configuration application:

        echo '
#!/bin/sh

APP_DIR="/data/simaai/applications"
APP_NAME="resnet50_sample_name"
ETC_PATH="${APP_DIR}/${APP_NAME}/etc"
PREPROC_CVU_CONFIG_BIN="genpreproc_200_cvu_cfg_app"
PREPROC_CVU_CONFIG_JSON="genpreproc_200_cvu_cfg_params.json"
DETESSDEQUANT_CVU_CONFIG_BIN="detessdequant_201_cvu_cfg_app"
DETESSDEQUANT_CVU_CONFIG_JSON="detessdequant_201_cvu_cfg_params.json"

# Execute preprocessing config
${ETC_PATH}/${PREPROC_CVU_CONFIG_BIN} ${ETC_PATH}/${PREPROC_CVU_CONFIG_JSON}

# Execute postprocessing config
${ETC_PATH}/${DETESSDEQUANT_CVU_CONFIG_BIN} ${ETC_PATH}/${DETESSDEQUANT_CVU_CONFIG_JSON}
' > resources/run_cvu_cfgs.sh

      • The new directory structure should look like this:

        sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -aL 4 plugins
plugins/
├── processcvu
│   ├── cfg
│   │   ├── detessdequant_201_cvu_cfg_params.json
│   │   └── genpreproc_200_cvu_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocesscvu.cpp
│   ├── gstsimaaiprocesscvu.h
│   └── README.md
├── processmla
│   ├── cfg
│   │   └── simaaiprocessmla_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocessmlacommon.h
│   ├── gstsimaaiprocessmla.cpp
│   ├── gstsimaaiprocessmla.h
│   ├── gstsimaaiprocessmlastandalone.cpp
│   ├── README.md
│   └── res
│       └── quantized_resnet50_stage1_mla.lm
└── simaaisrc
    ├── CMakeLists.txt
    ├── gstsimaaisrc.cc
    ├── gstsimaaisrc.h
    └── README.md

  4. Update the .project/pluginsInfo.json with the path to the new plugin along with a new generated uuidgen id.

    • SKIP. We have not added a new plugin, so need to update the .project/pluginsInfo.json file.

  5. Update the application.json with the new plugin’s name, pluginGid (same as previous step), and [resources][configs]:

    application.json
     1{
 2    "name": "resnet50_sample_name",
 3    "appId": "resnet50_sample_id",
 4    "sequence": 1,
 5    "pipelines": [
 6        {
 7            "name": "resnet50_sample_pipeline_name",
 8            "sequence": 1,
 9            "plugins": [
10                {
11                    "name": "simaaisrc",
12                    "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
13                    "resources": {
14                        "configs": []
15                    }
16                },
17                {
18                    "name": "processcvu",
19                    "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a",
20                    "resources": {
21                        "configs": [
22                            "cfg/genpreproc_200_cvu_cfg_params.json",
23                            "cfg/detessdequant_201_cvu_cfg_params.json"
24                        ]
25                    }
26                },
27                {
28                    "name": "processmla",
29                    "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d",
30                    "resources": {
31                        "configs": [
32                            "cfg/simaaiprocessmla_cfg_params.json"
33                        ],
34                        "lms": [
35                            "res/quantized_resnet50_stage1_mla.lm"
36                        ]
37                    },
38                    "dependencies": {
39                        "value": [
40                            "dispatcher-lite.zip"
41                        ]
42                    }
43                }
44            ],
45            "connections": [],
46            "resources": [
47                {
48                    "name": "genpreproc_200_cvu_cfg_app",
49                    "location": "resources/genpreproc_200_cvu_cfg_app",
50                    "destination": {
51                        "provider": "RPM",
52                        "type": "config"
53                    }
54                },
55                {
56                    "name": "detessdequant_201_cvu_cfg_app",
57                    "location": "resources/detessdequant_201_cvu_cfg_app",
58                    "destination": {
59                        "provider": "RPM",
60                        "type": "config"
61                    }
62                },
63                {
64                    "name": "run_cvu_cfgs_sh",
65                    "location": "resources/run_cvu_cfgs.sh",
66                    "destination": {
67                        "provider": "RPM",
68                        "type": "bin"
69                    }
70                }
71            ],
72            "dependencies": [
73                {
74                    "name": "run_cvu_cfgs_sh",
75                    "location": "/data/simaai/applications/resnet50_sample_name/bin/",
76                    "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh"
77                }
78            ]
79        }
80    ]
81}

    Note

    Let’s review the changes to the application.json:

    • Added ["pipelines"]["plugins"]["processcvu"]["resources"]: cfg/detessdequant_201_cvu_cfg_params.json:

      • Makes our CVU graph configuration file available to the simaaiprocesscvu plugin.

    • Added ["pipelines"]["resources"]["name"]: "detessdequant_201_cvu_cfg_app":

      • This makes our postprocessing CVU configuration binary available to our application.

Adding argmax_print custom plugin

Description

In this step we will be adding the custom plugin argmax_print that we developed and tested in:

Steps

  1. Copy the plugin from /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base to /home/docker/sima-cli/resnet50_app/plugins/:

    Note

    • For any custom plugin that uses the aggregator template, we also need to copy over:

      • /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/templates/ to the root of our application directory.

      • We then need to edit the target_include_directories in plugins/argmax_print/CMakeLists.txt from ${CMAKE_CURRENT_SOURCE_DIR}/../templates to ${CMAKE_CURRENT_SOURCE_DIR}/templates.

    sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/argmax_print/ plugins/
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cp -r /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/templates/ plugins/argmax_print/
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -L 2 plugins/
plugins/
├── argmax_print
│   ├── argmax_print_cfg.json
│   ├── CMakeLists.txt
│   ├── payload.cpp
│   └── templates
├── processcvu
│   ├── cfg
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocesscvu.cpp
│   ├── gstsimaaiprocesscvu.h
│   └── README.md
├── processmla
│   ├── cfg
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocessmlacommon.h
│   ├── gstsimaaiprocessmla.cpp
│   ├── gstsimaaiprocessmla.h
│   ├── gstsimaaiprocessmlastandalone.cpp
│   ├── README.md
│   └── res
└── simaaisrc
    ├── CMakeLists.txt
    ├── gstsimaaisrc.cc
    ├── gstsimaaisrc.h
    └── README.md

  2. Create a cfg/ directory:

    sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mkdir plugins/argmax_print/cfg
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -L 2 plugins/
plugins/
├── argmax_print
│   ├── argmax_print_cfg.json
│   ├── cfg
│   ├── CMakeLists.txt
│   ├── payload.cpp
│   └── templates
├── processcvu
│   ├── cfg
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocesscvu.cpp
│   ├── gstsimaaiprocesscvu.h
│   └── README.md
├── processmla
│   ├── cfg
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocessmlacommon.h
│   ├── gstsimaaiprocessmla.cpp
│   ├── gstsimaaiprocessmla.h
│   ├── gstsimaaiprocessmlastandalone.cpp
│   ├── README.md
│   └── res
└── simaaisrc
    ├── CMakeLists.txt
    ├── gstsimaaisrc.cc
    ├── gstsimaaisrc.h
    └── README.md

    Note

    The file argmax_print_cfg.json is the configuration file created when we developed the plugin. If you created it elsewhere, please locate it as we will need to copy it into the cfg directory.

  3. Create the configuration file inside the cfg/ directory:

    During the development of the argmax_print custom plugin, a configuration file was written. That file was brought in when we copied the plugin (argmax_print_cfg.json). Move the configuration file into the cfg/ directory:

    sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ mv plugins/argmax_print/argmax_print_cfg.json plugins/argmax_print/cfg
sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -L 2 plugins/argmax_print
plugins
├── argmax_print
│   ├── cfg
│   │   └── argmax_print_cfg.json
│   ├── CMakeLists.txt
│   ├── payload.cpp
│   └── templates
├── processcvu
│   ├── cfg
│   │   ├── detessdequant_201_cvu_cfg_params.json
│   │   └── genpreproc_200_cvu_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocesscvu.cpp
│   ├── gstsimaaiprocesscvu.h
│   └── README.md
├── processmla
│   ├── cfg
│   │   └── simaaiprocessmla_cfg_params.json
│   ├── CMakeLists.txt
│   ├── gstsimaaiprocessmlacommon.h
│   ├── gstsimaaiprocessmla.cpp
│   ├── gstsimaaiprocessmla.h
│   ├── gstsimaaiprocessmlastandalone.cpp
│   ├── README.md
│   └── res
│       └── quantized_resnet50_stage1_mla.lm
└── simaaisrc
    ├── CMakeLists.txt
    ├── gstsimaaisrc.cc
    ├── gstsimaaisrc.h
    └── README.md

  4. Update the .project/pluginsInfo.json with the path to the new plugin along with a new generated uuidgen id.

    • Generate a new uuid for this plugin:

      sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ uuidgen
b0e0910f-a58b-419f-a58d-584d94691836

    • Update the .project/pluginsInfo.json to:

       1{
 2    "pluginsInfo" : [
 3        {
 4            "gid" : "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
 5            "path" : "plugins/simaaisrc"
 6        },
 7        {
 8            "gid" : "09b09793-e1de-4881-9fd8-7674b948864a",
 9            "path" : "plugins/processcvu"
10        },
11        {
12            "gid" : "c086febf-7f33-447d-a4ed-96be718de36d",
13            "path" : "plugins/processmla"
14        },
15        {
16            "gid" : "b0e0910f-a58b-419f-a58d-584d94691836",
17            "path" : "plugins/argmax_print"
18        }
19    ]
20}

  5. Update the application.json with the new plugin’s name, pluginGid (same as previous step), and [resources][configs]:

    application.json
     1{
 2    "name": "resnet50_sample_name",
 3    "appId": "resnet50_sample_id",
 4    "sequence": 1,
 5    "pipelines": [
 6        {
 7            "name": "resnet50_sample_pipeline_name",
 8            "sequence": 1,
 9            "plugins": [
10                {
11                    "name": "simaaisrc",
12                    "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
13                    "resources": {
14                        "configs": []
15                    }
16                },
17                {
18                    "name": "processcvu",
19                    "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a",
20                    "resources": {
21                        "configs": [
22                            "cfg/genpreproc_200_cvu_cfg_params.json",
23                            "cfg/detessdequant_201_cvu_cfg_params.json"
24                        ]
25                    }
26                },
27                {
28                    "name": "processmla",
29                    "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d",
30                    "resources": {
31                        "configs": [
32                            "cfg/simaaiprocessmla_cfg_params.json"
33                        ],
34                        "lms": [
35                            "res/quantized_resnet50_stage1_mla.lm"
36                        ]
37                    },
38                    "dependencies": {
39                        "value": [
40                            "dispatcher-lite.zip"
41                        ]
42                    }
43                },
44                {
45                    "name": "argmax_print",
46                    "pluginGid": "b0e0910f-a58b-419f-a58d-584d94691836",
47                    "resources": {
48                        "configs": [
49                            "cfg/argmax_print_cfg.json"
50                        ]
51                    }
52                }
53            ],
54            "connections": [],
55            "resources": [
56                {
57                    "name": "genpreproc_200_cvu_cfg_app",
58                    "location": "resources/genpreproc_200_cvu_cfg_app",
59                    "destination": {
60                        "provider": "RPM",
61                        "type": "config"
62                    }
63                },
64                {
65                    "name": "detessdequant_201_cvu_cfg_app",
66                    "location": "resources/detessdequant_201_cvu_cfg_app",
67                    "destination": {
68                        "provider": "RPM",
69                        "type": "config"
70                    }
71                },
72                {
73                    "name": "run_cvu_cfgs_sh",
74                    "location": "resources/run_cvu_cfgs.sh",
75                    "destination": {
76                        "provider": "RPM",
77                        "type": "bin"
78                    }
79                }
80            ],
81            "dependencies": [
82                {
83                    "name": "run_cvu_cfgs_sh",
84                    "location": "/data/simaai/applications/resnet50_sample_name/bin/",
85                    "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh"
86                }
87            ]
88        }
89    ]
90}

    Note

    Let’s review the changes to the application.json:

    • Added ["pipelines"]["plugins"]["name"]: "argmax_print" with its corresponding uuid and configuration file.

Adding the GStreamer Command

Now that we have defined our application plugins, resources and configurations, it is time to provide the runtime string that will run when the application is deployed.

Similarly like we did when using the standard gstreamer gst-launch-1.0 command, the gst-launch command will look like:

gst-launch-1.0 --gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib' -v \
simaaisrc mem-target=1 node-name=\"my_image_src\" location=\"/data/simaai/applications/resnet50_sample_name/etc/golden_retriever_207_rgb.bin\" num-buffers=1 ! \
simaaiprocesscvu source-node-name=\"my_image_src\" buffers-list=\"my_image_src\" config=\"/data/simaai/applications/resnet50_sample_name/etc/genpreproc_200_cvu_cfg_params.json\" ! \
simaaiprocessmla config=\"/data/simaai/applications/resnet50_sample_name/etc/simaaiprocessmla_cfg_params.json\" ! \
simaaiprocesscvu source-node-name=\"mla-resnet\" buffers-list=\"mla-resnet\" config=\"/data/simaai/applications/resnet50_sample_name/etc/detessdequant_201_cvu_cfg_params.json\" name=\"detessdequant\" ! \
argmax_print config=\"/data/simaai/applications/resnet50_sample_name/etc/argmax_print_cfg.json\" ! \
fakesink"

Note

In reading the gst-launch string above, there is one notable difference from the one used in Developing & Debugging GStreamer Applications: * The paths are all different. They will depend on where our application is installed and where we have placed our resources relative to that path.

Lastly, in JSON we cannot add a multi-line string, so in the application.json below the command will be written as a single line. It has been split above into multiple lines for readability.

application.json
 1{
 2    "name": "resnet50_sample_name",
 3    "appId": "resnet50_sample_id",
 4    "sequence": 1,
 5    "pipelines": [
 6        {
 7            "name": "resnet50_sample_pipeline_name",
 8            "sequence": 1,
 9            "plugins": [
10                {
11                    "name": "simaaisrc",
12                    "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
13                    "resources": {
14                        "configs": []
15                    }
16                },
17                {
18                    "name": "processcvu",
19                    "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a",
20                    "resources": {
21                        "configs": [
22                            "cfg/genpreproc_200_cvu_cfg_params.json",
23                            "cfg/detessdequant_201_cvu_cfg_params.json"
24                        ]
25                    }
26                },
27                {
28                    "name": "processmla",
29                    "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d",
30                    "resources": {
31                        "configs": [
32                            "cfg/simaaiprocessmla_cfg_params.json"
33                        ],
34                        "lms": [
35                            "res/quantized_resnet50_stage1_mla.lm"
36                        ]
37                    },
38                    "dependencies": {
39                        "value": [
40                            "dispatcher-lite.zip"
41                        ]
42                    }
43                },
44                {
45                    "name": "argmax_print",
46                    "pluginGid": "b0e0910f-a58b-419f-a58d-584d94691836",
47                    "resources": {
48                        "configs": [
49                            "cfg/argmax_print_cfg.json"
50                        ]
51                    }
52                }
53            ],
54            "connections": [],
55            "resources": [
56                {
57                    "name": "genpreproc_200_cvu_cfg_app",
58                    "location": "resources/genpreproc_200_cvu_cfg_app",
59                    "destination": {
60                        "provider": "RPM",
61                        "type": "config"
62                    }
63                },
64                {
65                    "name": "detessdequant_201_cvu_cfg_app",
66                    "location": "resources/detessdequant_201_cvu_cfg_app",
67                    "destination": {
68                        "provider": "RPM",
69                        "type": "config"
70                    }
71                },
72                {
73                    "name": "run_cvu_cfgs_sh",
74                    "location": "resources/run_cvu_cfgs.sh",
75                    "destination": {
76                        "provider": "RPM",
77                        "type": "bin"
78                    }
79                }
80            ],
81            "dependencies": [
82                {
83                    "name": "run_cvu_cfgs_sh",
84                    "location": "/data/simaai/applications/resnet50_sample_name/bin/",
85                    "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh"
86                }
87            ],
88            "gst": "gst-launch-1.0 --gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib' -v simaaisrc mem-target=1 node-name=\"my_image_src\" location=\"/data/simaai/applications/resnet50_sample_name/etc/golden_retriever_207_rgb.bin\" num-buffers=1 ! simaaiprocesscvu source-node-name=\"my_image_src\" buffers-list=\"my_image_src\" config=\"/data/simaai/applications/resnet50_sample_name/etc/genpreproc_200_cvu_cfg_params.json\" ! simaaiprocessmla config=\"/data/simaai/applications/resnet50_sample_name/etc/simaaiprocessmla_cfg_params.json\" ! simaaiprocesscvu source-node-name=\"mla-resnet\" buffers-list=\"mla-resnet\" config=\"/data/simaai/applications/resnet50_sample_name/etc/detessdequant_201_cvu_cfg_params.json\" name=\"detessdequant\" ! argmax_print config=\"/data/simaai/applications/resnet50_sample_name/etc/argmax_print_cfg.json\" ! fakesink"
89        }
90    ]
91}

Adding Final Configurations and Resources

In this section, we will add two additional items to our application.json:abbr:

  1. The input to the pipeline that we have been using during testing: golden_retriever_207_rgb.bin.

  2. Configuration information required by the mpk build system.

First, copy the golden_retriever_207_rgb.bin to the application resources directory. Directory should look like:

sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ tree -L 2
.
├── application.json
├── core
│   ├── allocator
│   ├── CMakeLists.txt
│   ├── dispatcher-lite
│   └── simamm
├── plugins
│   ├── argmax_print
│   ├── processcvu
│   ├── processmla
│   └── simaaisrc
├── project.mpk
└── resources
    ├── detessdequant_201_cvu_cfg_app
    ├── genpreproc_200_cvu_cfg_app
    ├── golden_retriever_207_rgb.bin
    └── run_cvu_cfgs.sh

Second, update the application.json:

application.json
  1{
  2    "name": "resnet50_sample_name",
  3    "appId": "resnet50_sample_id",
  4    "sequence": 1,
  5    "pipelines": [
  6        {
  7            "name": "resnet50_sample_pipeline_name",
  8            "sequence": 1,
  9            "plugins": [
 10                {
 11                    "name": "simaaisrc",
 12                    "pluginGid": "0c255b9a-e5a8-4346-993f-e3527cc6d4e3",
 13                    "resources": {
 14                        "configs": []
 15                    }
 16                },
 17                {
 18                    "name": "processcvu",
 19                    "pluginGid": "09b09793-e1de-4881-9fd8-7674b948864a",
 20                    "resources": {
 21                        "configs": [
 22                            "cfg/genpreproc_200_cvu_cfg_params.json",
 23                            "cfg/detessdequant_201_cvu_cfg_params.json"
 24                        ]
 25                    }
 26                },
 27                {
 28                    "name": "processmla",
 29                    "pluginGid": "c086febf-7f33-447d-a4ed-96be718de36d",
 30                    "resources": {
 31                        "configs": [
 32                            "cfg/simaaiprocessmla_cfg_params.json"
 33                        ],
 34                        "lms": [
 35                            "res/quantized_resnet50_stage1_mla.lm"
 36                        ]
 37                    },
 38                    "dependencies": {
 39                        "value": [
 40                            "dispatcher-lite.zip"
 41                        ]
 42                    }
 43                },
 44                {
 45                    "name": "argmax_print",
 46                    "pluginGid": "b0e0910f-a58b-419f-a58d-584d94691836",
 47                    "resources": {
 48                        "configs": [
 49                            "cfg/argmax_print_cfg.json"
 50                        ]
 51                    }
 52                }
 53            ],
 54            "connections": [],
 55            "resources": [
 56                {
 57                    "name": "genpreproc_200_cvu_cfg_app",
 58                    "location": "resources/genpreproc_200_cvu_cfg_app",
 59                    "destination": {
 60                        "provider": "RPM",
 61                        "type": "config"
 62                    }
 63                },
 64                {
 65                    "name": "detessdequant_201_cvu_cfg_app",
 66                    "location": "resources/detessdequant_201_cvu_cfg_app",
 67                    "destination": {
 68                        "provider": "RPM",
 69                        "type": "config"
 70                    }
 71                },
 72                {
 73                    "name": "run_cvu_cfgs_sh",
 74                    "location": "resources/run_cvu_cfgs.sh",
 75                    "destination": {
 76                        "provider": "RPM",
 77                        "type": "bin"
 78                    }
 79                },
 80                {
 81                "name": "golden_retriever_207_rgb.bin",
 82                "location": "resources/golden_retriever_207_rgb.bin",
 83                "destination": {
 84                    "provider": "RPM",
 85                    "type": "config"
 86                }
 87                }
 88            ],
 89            "dependencies": [
 90                {
 91                    "name": "run_cvu_cfgs_sh",
 92                    "location": "/data/simaai/applications/resnet50_sample_name/bin/",
 93                    "command": "sh /data/simaai/applications/resnet50_sample_name/bin/run_cvu_cfgs.sh"
 94                }
 95            ],
 96            "gst": "gst-launch-1.0 --gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib' -v simaaisrc mem-target=1 node-name=\"my_image_src\" location=\"/data/simaai/applications/resnet50_sample_name/etc/golden_retriever_207_rgb.bin\" num-buffers=1 ! simaaiprocesscvu source-node-name=\"my_image_src\" buffers-list=\"my_image_src\" config=\"/data/simaai/applications/resnet50_sample_name/etc/genpreproc_200_cvu_cfg_params.json\" ! simaaiprocessmla config=\"/data/simaai/applications/resnet50_sample_name/etc/simaaiprocessmla_cfg_params.json\" ! simaaiprocesscvu source-node-name=\"mla-resnet\" buffers-list=\"mla-resnet\" config=\"/data/simaai/applications/resnet50_sample_name/etc/detessdequant_201_cvu_cfg_params.json\" name=\"detessdequant\" ! argmax_print config=\"/data/simaai/applications/resnet50_sample_name/etc/argmax_print_cfg.json\" ! fakesink"
 97        }
 98    ],
 99    "configuration": {
100        "installationPrefixes": {
101            "configurations": "/data/simaai/applications/resnet50_sample_name/etc",
102            "binaries": "/data/simaai/applications/resnet50_sample_name/bin",
103            "resources": "/data/simaai/applications/resnet50_sample_name/share",
104            "libraries": "/data/simaai/applications/resnet50_sample_name/lib",
105            "globalLibs": "/usr/lib"
106        },
107        "compilation": {
108            "copyPokyLibs": false
109        },
110        "gst": {
111            "options": [
112                "--gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib'"
113            ]
114        },
115        "environment": [
116            "GST_DEBUG=0",
117            "LD_LIBRARY_PATH='/data/simaai/applications/resnet50_sample_name/lib'"
118        ]
119    },
120    "manifestVersion": "2.0.0"
121}

Note

Let’s review the changes to the application.json:

  • Added ["configuration"]["installationPrefixes"]["configurations" | "binaries" | "resources" | "libraries" | "globalLibs"]

    • This is the main paths where we will store various files and resources for our application. Standard to leave as shown here but change the name of the application.

  • Added ["configuration"]["compilation"]["copyPokyLibs"]: false

    • This is telling the system that the necessary libs are already on pre-built on the board and do not need to be copied.

  • Added ["configuration"]["gst"]["options"]: "--gst-plugin-path='/data/simaai/applications/resnet50_sample_name/lib'"

    • This is the location of where our plugins will be copied to by the mpk build system.

  • Added ["configuration"]["environment"]["GST_DEBUG" | "LD_LIBRARY_PATH"]:

    • These are any environment variables we want to set during our application launch.

  • Added ["manifestVersion"]: 2.0.0

    • The version of the application.json manifest, and how it will be interpreted. For this version of the guide, always use this manifest version in all of your applications.

Create the MPK

Now that we have fully tested and defined our application, we are ready to package it into an mpk file.

To do so we will use the command mpk create as referenced in MPK Tool. Simply step one directory our of the application project directory and invoke the command:

sima-user@docker-image-id:/home/docker/sima-cli/resnet50_app$ cd ..
sima-user@docker-image-id:/home/docker/sima-cli$ mpk create -s resnet50_app/ -d resnet50_app/ --clean
ℹ Cleaning up build artifacts...
✔ Successfully cleaned up build artifacts.
ℹ Step a65-apps COMPILE completed successfully.
ℹ Step COMPILE completed successfully.
ℹ Step COPY RESOURCE completed successfully
ℹ Step RPM BUILD completed successfully.
✔ Successfully created MPK at '/home/docker/sima-cli/resnet50_app/project.mpk'
ℹ Cleaning up build artifacts...
✔ Successfully cleaned up build artifacts.
sima-user@docker-image-id:/home/docker/sima-cli$ tree -L 1 resnet50_app/
resnet50_app/
├── application.json
├── core
├── plugins
├── project.mpk
└── resources

The project.mpk is the packaged up application that is ready to be installed. You can rename this if you wish.

Deploy the MPK

Lastly, in order to deploy our application to our device, we will make use of the command mpk deploy as referenced in MPK Tool.

Note

In order to perform this step, make sure you have used mpk device connect and currently have a device successfully connected.

sima-user@docker-image-id:/home/docker/sima-cli$ mpk device list

                        Active Connections over Ethernet
┏━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Status ┃ Status Details ┃   Target   ┃ Username ┃    Device    ┃ Jump Server ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│   🟢   │     Online     │ 10.0.0.143 │   sima   │ fence_wonder │      -      │
└────────┴────────────────┴────────────┴──────────┴──────────────┴─────────────┘
❕ There are no PCIe devices available.
sima-user@docker-image-id:/home/docker/sima-cli$ mpk deploy -d fence_wonder -f resnet50_app/project.mpk
🚀 Sending MPK to 10.0.0.143...
Transfer Progress for project.mpk:  100.00%
🏁 MPK sent successfully!
✔ MPK Deployed! ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
✔ MPK Deployment is successful for project.mpk.

At this point, we have deployed the mpk successfully on the board, and the application should have executed right away. We can verify that the application executed as expected by checking the logs.

On the MLSoC device:

  1. Navigate to /tmp/resnet50_sample_pipeline_name (name of our pipeline in the application.json)

  2. Print the gst-launch output:

    davinci:/tmp/resnet50_sample_pipeline_name$ cat gst-launch-1.0
...
Index of largest value: 207
Largest value: 0.959188
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
Redistribute latency...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.001224352
Setting pipeline to NULL ...
Freeing pipeline ...

At this point we have learned how to develop, test, debug, and package an application target to an MLSoC device. Future examples and application notes will contain more advanced applications, but all of the essential concepts have been covered in this guide.