Build with Palette

This step-by-step guide walks you through building the ResNet-50 pipeline using Palette.

Note

This article is mainly for the standalone mode, but it is also useful to go through for the PCIe mode, as the core principles of pipeline development remain the same.

For PCIe mode, you will need to build additional software on the host side, using the PCIe driver and APIs to interact with the board. Once you completed this guide, you can refer to this page to learn how to adapt this pipeline to run on a PCIe system.

Step 1. Install and Setup Palette

Note

Download Palette

You will need a modern Ubuntu 22.04+ or Windows 11 Pro machine to run Palette. For more information on system requirements and installation procedure, refer to Software Installation. Make sure your DevKit runs v1.5 firmware or above, click here to find out how to check firmware version and update if necessary.

The Palette container maps the host file system. You can create and edit the Python script in the host machine using your favorite IDE. The files will be accessible in the Palette environment. To find out how the host file system is mapped to Palette, run this command below.

user@sima-user-machine:~/<PALETTE_VERSION>/sima-cli$ command -v jq >/dev/null 2>&1 || { echo "jq not found. Installing..."; sudo apt update && sudo apt install -y jq; } && docker inspect $(docker ps -q --filter "name=palettesdk") --format='{{json .Mounts}}' | jq
{
    "Type": "bind",
    "Source": "/home/user/workspace",
    "Destination": "/home/docker/sima-cli",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
}

The Source element shows the host path, mapping to /home/docker/sima-cli in the Palette environment.

Step 2. Create a New Project

First, download the optimized ResNet50 model. To learn more on how to optimize a standard Resnet50 ONNX model refer to this link.

Next, download the input resource image. Then copy both files to a folder accessible by the Palette container environment.

Once the files are in place, run the following command inside the Palette environment to create a new project: mpk project create.

Note

Please note this procedure only supports BitMap image format, if you have a dataset that is in JPEG or other format, please make sure you convert them to BitMap format first before proceeding.

user@palette-container-id:/home/docker/sima-cli/demo$ mpk project create --model-path resnet_50.tar.gz --input-resource input_image720.rgb
β„Ή No plugin configuration file provided. Creating full pipeline corresponding to model resnet_50
β„Ή Step: Base directory setup at: /home/docker/sima-cli/demo/resnet_50 completed successfully
β„Ή Step: Artifacts for gst_app copied successfully.
β„Ή Step: Plugin artifacts for simaaisrc copied successfully.
β„Ή Step: Plugin artifacts for detesdequant copied successfully.
β„Ή Step: Plugin artifacts for process_mla copied successfully.
β„Ή Step: Plugin artifacts for gen_preproc copied successfully.
β„Ή Step: .project/pluginsInfo.json successfully.
β„Ή Step: application.json file created successfully.
β„Ή Step: Project 'resnet_50' has been successfully created at /home/docker/sima-cli/test.

This command automatically generates the project skeleton based on the embedded information within the optimized model. To learn more on how this Resnet50 model was optimized for SiMa MLA, refer to this link.

Step 3. Understand the Project Structure

The skeleton created in the previous step includes an application.json pipeline definition file and a number of plugin files. Understanding the structure of this file is crucial, as we need to modify it to meet our requirements. The application.json file defines the GStreamer inferencing pipeline and provides a gst-launch command. When the app is packaged and deployed, the SiMa MLA interprets this file to start the appropriate components accordingly.

user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ tree -L 2
.
β”œβ”€β”€ application.json
β”œβ”€β”€ dependencies
β”‚Β Β  └── gst_app
└── plugins
    β”œβ”€β”€ detesdequant
    β”œβ”€β”€ gen_preproc
    β”œβ”€β”€ process_mla
    └── simaaisrc

7 directories, 1 file

To help you understand the structure of the application.json file, you can use this visualization tool and click through the automatically generated nodes in the pipeline to see the details of each component.

View application.json

Step 4. About the Data Input

To faciliate with file operations, SiMa provides the simaaisrc plugin. This plugin operates similarly to the standard filesrc and multifilesrc plugins but optimized for SiMa MLA. It serves as a source element that reads data from files and feeds it into a GStreamer pipeline. This functionality is particularly useful for testing and debugging, as it allows developers to simulate various input scenarios by sourcing data from files, thereby facilitating isolated testing of individual components within a pipeline.

This plugin only supports RGB format binary files, so if you want to pass a JPEG image to this plugin you must convert it to RGB format first. For your convenience we provided an example code for image conversion.

View Sample Code for Resizing and Reformattng Images
import cv2
import numpy as np
import glob
import os
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s')

VALID_EXTENSIONS = {'.jpg', '.jpeg', '.png', '.bmp', '.tiff', '.tif'}

def process_images(target_count=100, source_dir='.', output_dir='raw_data'):
    os.makedirs(output_dir, exist_ok=True)
    image_files = [f for ext in VALID_EXTENSIONS for f in glob.glob(f"{source_dir}/*{ext}")]

    if not image_files:
        logging.warning("No images found")
        return

    for i, img_path in enumerate(image_files[:target_count]):
        try:
            img = cv2.imread(img_path)
            if img is None:
                logging.error("Failed to read: %s", img_path)
                continue

            img_resized = cv2.resize(img, (1280, 720), interpolation=cv2.INTER_CUBIC)
            output_path = os.path.join(output_dir, f"{i+1}.out")
            img_resized.tofile(output_path)
            logging.info("Processed %s -> %s", os.path.basename(img_path), output_path)

        except Exception as e:
            logging.error("Error with %s: %s", img_path, str(e))

    logging.info("Generated %d/%d images", min(len(image_files), target_count), target_count)

if __name__ == "__main__":
    process_images()
Step 5. Add a Custom Postprocessing Plugin

By default, when the project skeleton was created, the pipeline terminates at fakesink which means the output goes no where. To check the inference output we need to write a post processing plugin and place into the pipeline.

Let take a shortcut and bring an example plugin from the SDK /usr/local/simaai/app_zoo/GStreamer/GenericAggregatorDemo/plugins/simple_overlay into the project. We also need to copy the template headers from the SDK into the project.

user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ cp -R /usr/local/simaai/app_zoo/Gstreamer/GenericAggregatorDemo/plugins/simple_overlay/ plugins
user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ cp -R /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/templates/ plugins/

Then, let’s edit the application.json file and replace the fakesink to the simple_overlay plugin. We also need to change the hidden file .project/pluginsInfo.json.

user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ sed -i 's/fakesink/simple_overlay/g' application.json
user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ sed -i 's/fakesink/simple_overlay/g' .project/pluginsInfo.json

We also need to modify the plugin configuration file plugins/simple_overlay/overlay.json to include the proper caps negotiation for the input of the plugin.

"caps": {
"sink_pads": [
{
    "media_type": "application/vnd.simaai.tensor",
    "params": [
    {
        "name": "format",
        "type": "string",
        "values": "DETESSDEQUANT",
        "json_field": null
    }
    ]
}
],
Step 6. Modify the gst String

The gst string in the application.json file defines the gStreamer launch command. We need to modify this string to include the new post processing plugin.

Original string:

"gst": "simaaisrc location=/data/simaai/applications/resnet_50/etc/input_image720.rgb node-name=input delay=1 mem-target=0 index=0 loop=false ! 'video/x-raw, format=(string)RGB, width=(int)1920, height=(int)1080' ! simaaiprocesscvu name=simaai_gen_preproc_1 !  simaaiprocessmla name=simaai_process_mla_1 ! simaaiprocesscvu name=simaai_detesdequant_1 ! 'application/vnd.simaai.tensor, format=(string)DETESSDEQUANT' ! fakesink"

New string:

"gst": "simaaisrc location=/data/simaai/applications/resnet_50/etc/input_image720.rgb node-name=input delay=1 mem-target=0 index=0 loop=false ! 'video/x-raw, format=(string)RGB, width=(int)1280, height=(int)720' ! simaaiprocesscvu name=simaai_gen_preproc_1 !  simaaiprocessmla name=simaai_process_mla_1 ! simaaiprocesscvu name=simaai_detesdequant_1 ! 'application/vnd.simaai.tensor, format=(string)DETESSDEQUANT' ! simple_overlay config=/data/simaai/applications/resnet_50/etc/overlay.json"

The changes in the new strings are:

  • width and height in the simaaisrc plugin is changed to 1280 and 720 respectively.

  • Added config parameter to the simple_overlay plugins.

Step 7. Add Configuration File to the Custom Plugin

In order for the compilation process to include the configuration file for the custom plugin, it must also be declared explicitly in the application.json file. Replace the resources block in the simple_overlay block in application.json as below.

{
    ... ... ...
    "resources": {"configs": ["cfg/overlay.json"]}
}

After the changes the application.json file should look like this:

View application.json (changed)

Step 8. Understand Resnet50 Output

ResNet-50 is typically trained on ImageNet (1000 classes), and its output is a 1D tensor (vector) of size 1000, representing the classification probabilities for each class.

Output Shape

(1, 1000)  # (Batch Size, Number of Classes)

Each value in this vector represents the probability score for a class. For example:

[0.001, 0.003, ..., 0.92, ..., 0.0005]  # 1000 values

  • The index with the highest value is the predicted class.

  • Example: If index 340 = 0.92, then the image is classified as zebra.

The plugin detesdequant is configured to output NHWC format, which stands for:

  • N β†’ Batch size (number of images processed together)

  • H β†’ Height (number of rows in the image)

  • W β†’ Width (number of columns in the image)

  • C β†’ Channels (number of color channels, e.g., 3 for RGB or 1 for grayscale)

Step 9. Add Custom Logic to Process Resnet50 Output

Once you understand the output of the detesdequant plugin, you can write a custom plugin to process the output. Use the following code to replace the UserContext::run function in the simple_overlay/payload.cpp

int argmax(float *probabilities, std::size_t size) {
    if (size == 0) return -1;
    int max_idx = 0;
    float max_value = probabilities[0];

    for (std::size_t i = 1; i < size; i++) {
        if (probabilities[i] > max_value) {
            max_value = probabilities[i];
            max_idx = i;
        }
    }
    return max_idx;
}

void UserContext::run(std::vector<Input> &input,
                    std::span<uint8_t> output) {
    // Extract the input probabilities
    float *probabilities = reinterpret_cast<float*>(input[0].getData().data());
    std::size_t probabilities_size = input[0].getDataSize();

    // Ensure valid input data
    if (probabilities == nullptr || probabilities_size == 0) {
        std::cerr << "Error: Invalid input data." << std::endl;
        return;
    }

    // Convert byte size to element count
    std::size_t num_elements = probabilities_size / sizeof(float);
    if (num_elements == 0) {
        std::cerr << "Error: No valid data in input." << std::endl;
        return;
    }

    // Perform the argmax operation
    int max_idx = argmax(probabilities, num_elements);

    // Print the class index and its probability
    std::cout << "Predicted Class Index: " << max_idx << std::endl;
    std::cout << "Probability: " << probabilities[max_idx] << std::endl;
}

The provided C++ code above processes the output of the detesdequant plugin and determines the most probable class based on its probability distribution.

argmax Function

  • Iterates through an array of floating-point probabilities.

  • Finds the index of the highest probability value (i.e., the most confident prediction).

  • Returns the index of the class with the highest probability.

UserContext::run Function

  • Extracts the probability tensor from the input.

  • Validates the input to ensure it is not empty.

  • Computes the most probable class using argmax().

  • Prints the predicted class index and its probability.

Step 10. Compile and Deploy the Project

From the Palette command line, execute the following commands to connect to the DevKit, build the MPK app package, deploy and execute the app.

user@palette-container-id:{project-folder}$ mpk device connect -t sima@<DevKit_IP_ADDRESS>

user@palette-container-id:{project-folder}$ mpk create -s . -d . --clean
β„Ή Compiling a65-apps...
βœ” a65-apps compiled successfully.
β„Ή Compiling Plugins...
βœ” Plugins Compiled successfully.
β„Ή Copying Resources...
βœ” Resources Copied successfully.
β„Ή Building Rpm...
βœ” Rpm built successfully.
β„Ή Creating mpk file...
βœ” Mpk file created successfully at {project-folder}/project.mpk .

user@palette-container-id:{project-folder}$ mpk deploy -f project.mpk
πŸš€ Sending MPK to <DevKit IP>...
Transfer Progress for project.mpk:  100.00%
🏁 MPK sent successfully!
βœ” MPK Deployed! ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
βœ” MPK Deployment is successful for project.mpk.

Note

You only need to run the mpk device connect once unless you shutdown the DevKit for extended period of time, or restarted the Palette container.

To connect to a PCIe device, use mpk device connect -s 0 where 0 stands for the slot number where the PCIe card is installed.

Step 11. Check the Runtime Output

Note

Connect to the DevKit console using SSH. For PCIe mode, connect through the virtual ethernet interface as described here.

When the pipeline is running, you can use the following command to check the running gst process’s full command line.

davinci:/var/log$ cat /proc/$(pgrep -f gst_app)/cmdline | tr '\0' ' '
/data/simaai/applications/resnet_50/bin/gst_app --manifest-json=/data/simaai/applications/resnet_50/manifest.json --gst-string=simaaisrc location=/data/simaai/applications/resnet_50/etc/input_image720.rgb node-name=input blocksize=2764800 delay=1 mem-target=0 ! 'video/x-raw, format=(string)RGB, width=(int)1280, height=(int)720' ! simaaiprocesscvu_new name=simaai_gen_preproc_1 config=/data/simaai/applications/resnet_50/etc/preproc.json !  simaaiprocessmla_new name=simaai_process_mla_1 config=/data/simaai/applications/resnet_50/etc/mla.json ! simaaiprocesscvu_new name=simaai_detesdequant_1 config=/data/simaai/applications/resnet_50/etc/detess_dequant.json ! 'application/vnd.simaai.tensor, format=(string)DETESSDEQUANT' ! simple_overlay config=/data/simaai/applications/resnet_50/etc/overlay.json davinci:/var/log$

If you want to check the process console log, you can tail /tmp/simaai/resnet_50_Pipeline/resnet50_mpk_Pipeline.1/gst_app.log:

davinci:/var/log$ tail -f /tmp/simaai/resnet_50_Pipeline/resnet50_mpk_Pipeline.1/gst_app.log
Predicted Class Index: 867
Probability: 0.435257
Predicted Class Index: 867
Probability: 0.435257
Predicted Class Index: 867
Probability: 0.435257
Predicted Class Index: 867
Probability: 0.435257
Predicted Class Index: 867
Probability: 0.435257
Predicted Class Index: 867
Probability: 0.435257
Predicted Class Index: 867
Probability: 0.435257
Predicted Class Index: 867
Probability: 0.435257
Predicted Class Index: 867
Probability: 0.435257

To run the pipeline directly from the DevKit, first stop the running process, then execute the command below.

This approach offers greater flexibility for debugging the pipeline without relying on the MPK process. It also demonstrates how the pipeline can be integrated into a larger application in standalone mode. For example, you can add the command to the DevKit’s startup script to automatically run the pipeline at boot.

davinci:/var/log$ sudo kill $(pgrep -f gst_app)
davinci:/var/log$ sudo GST_DEBUG=3 LD_LIBRARY_PATH=/data/simaai/applications/resnet_50/lib GST_PLUGIN_PATH=/data/simaai/applications/resnet_50/lib /data/simaai/applications/resnet_50/bin/gst_app --manifest-json=/data/simaai/applications/resnet_50/manifest.json --gst-string="simaaisrc location=/data/simaai/applications/resnet_50/etc/input_image720.rgb node-name=input delay=1 mem-target=0 index=0 loop=false ! 'video/x-raw, format=(string)RGB, width=(int)1280, height=(int)720' ! simaaiprocesscvu name=simaai_gen_preproc_1 !  simaaiprocessmla name=simaai_process_mla_1 ! simaaiprocesscvu name=simaai_detesdequant_1 ! 'application/vnd.simaai.tensor, format=(string)DETESSDEQUANT' ! simple_overlay config=/data/simaai/applications/resnet_50/etc/overlay.json"

The command is constructed with the following parts:

  • GST_DEBUG: standard gstreamer debug level environment.

  • LD_LIBRARY_PATH: path to the library.

  • GST_PLUGIN_PATH: path to the plugin.

  • /data/simaai/applications/resnet_50/bin/gst_app: the path to the gst_app binary.

  • --manifest-json: the path to the manifest.json file.

  • --gst-string: the gstreamer pipeline string. You can get this string from the manifest.json file, or using the command mentioned in the beginning of this step to get the full cmdline.