.. _building_first_pipeline_with_palette: ================== Build with Palette ================== This step-by-step guide walks you through building the ResNet-50 pipeline using :ref:`Palette CLI`. .. note:: This article is mainly for the `standalone mode <../target_architectures.html#sima-devkit-as-a-standalone-device>`_, but it is also useful to go through for the `PCIe mode <../target_architectures.html#sima-devkit-as-pcie-card>`_, as the core principles of pipeline development remain the same. For PCIe mode, you will need to build additional software on the host side, using the PCIe driver and :ref:`APIs ` to interact with the board. Once you completed this guide, you can refer to this :ref:`page ` to learn how to adapt this pipeline to run on a PCIe system. .. dropdown:: Step 1. Install and Setup Palette :animate: fade-in :color: secondary .. note:: .. button-link:: https://docs.sima.ai/pkg_downloads/SDK1.6.0/1.6.0_Palette_SDK_master_B202.zip :color: primary :shadow: Download Palette You will need a modern Ubuntu 22.04+ or Windows 11 Pro machine to run Palette. For more information on system requirements and installation procedure, refer to :ref:`Palette installation`. Make sure your DevKit runs v1.5 firmware or above, click :ref:`here ` to find out how to check firmware version and update if necessary. .. include:: ../blocks/palette_volume_mapping.rst .. dropdown:: Step 2. Create a New Project :animate: fade-in :color: secondary First, download the optimized `ResNet50 model `_. To learn more on how to optimize a standard Resnet50 ONNX model refer to this :ref:`link `. Next, download the `input resource image `_. Then copy both files to a folder accessible by the Palette container environment. Once the files are in place, run the following command inside the Palette environment to `create <../../palette/mpk_tools.html#mpk-project-create>`_ a new project: ``mpk project create``. .. note:: Please note this procedure only supports BitMap image format, if you have a dataset that is in JPEG or other format, please make sure you convert them to BitMap format first before proceeding. .. code-block:: console user@palette-container-id:/home/docker/sima-cli/demo$ mpk project create --model-path resnet_50.tar.gz --input-resource input_image720.rgb ℹ No plugin configuration file provided. Creating full pipeline corresponding to model resnet_50 ℹ Step: Base directory setup at: /home/docker/sima-cli/demo/resnet_50 completed successfully ℹ Step: Artifacts for gst_app copied successfully. ℹ Step: Plugin artifacts for simaaisrc copied successfully. ℹ Step: Plugin artifacts for detesdequant copied successfully. ℹ Step: Plugin artifacts for process_mla copied successfully. ℹ Step: Plugin artifacts for gen_preproc copied successfully. ℹ Step: .project/pluginsInfo.json successfully. ℹ Step: application.json file created successfully. ℹ Step: Project 'resnet_50' has been successfully created at /home/docker/sima-cli/test. This command automatically generates the project skeleton based on the embedded information within the optimized model. To learn more on how this Resnet50 model was optimized for SiMa MLA, refer to this :ref:`link `. .. dropdown:: Step 3. Understand the Project Structure :animate: fade-in :color: secondary The skeleton created in the previous step includes an ``application.json`` pipeline definition file and a number of plugin files. Understanding the structure of this file is crucial, as we need to modify it to meet our requirements. The ``application.json`` file defines the GStreamer inferencing pipeline and provides a gst-launch command. When the app is packaged and deployed, the SiMa MLA interprets this file to start the appropriate components accordingly. .. code:: console user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ tree -L 2 . ├── application.json ├── dependencies │   └── gst_app └── plugins ├── detesdequant ├── gen_preproc ├── process_mla └── simaaisrc 7 directories, 1 file To help you understand the structure of the ``application.json`` file, you can use this visualization tool and click through the automatically generated nodes in the pipeline to see the details of each component. .. button-link:: ../../../_static/pipeline-visualizer/app.html?data=hellosima-resnet50-palette.json :color: info :shadow: View application.json .. dropdown:: Step 4. About the Data Input :animate: fade-in :color: secondary To faciliate with file operations, SiMa provides the :ref:`simaaisrc plugin `. This plugin operates similarly to the standard filesrc and multifilesrc plugins but optimized for SiMa MLA. It serves as a source element that reads data from files and feeds it into a GStreamer pipeline. This functionality is particularly useful for testing and debugging, as it allows developers to simulate various input scenarios by sourcing data from files, thereby facilitating isolated testing of individual components within a pipeline. This plugin only supports RGB format binary files, so if you want to pass a JPEG image to this plugin you must convert it to RGB format first. For your convenience we provided an example code for image conversion. .. dropdown:: View Sample Code for Resizing and Reformattng Images :animate: fade-in :color: info .. include:: ../blocks/prepare_images.rst .. dropdown:: Step 5. Add a Custom Postprocessing Plugin :animate: fade-in :color: secondary By default, when the project skeleton was created, the pipeline terminates at ``fakesink`` which means the output goes no where. To check the inference output we need to write a post processing plugin and place into the pipeline. Let take a shortcut and bring an example plugin from the SDK ``/usr/local/simaai/app_zoo/GStreamer/GenericAggregatorDemo/plugins/simple_overlay`` into the project. We also need to copy the template headers from the SDK into the project. .. code:: console user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ cp -R /usr/local/simaai/app_zoo/Gstreamer/GenericAggregatorDemo/plugins/simple_overlay/ plugins user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ cp -R /usr/local/simaai/plugin_zoo/gst-simaai-plugins-base/gst/templates/ plugins/ Then, let's edit the ``application.json`` file and replace the ``fakesink`` to the ``simple_overlay`` plugin. We also need to change the hidden file ``.project/pluginsInfo.json``. .. code:: console user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ sed -i 's/fakesink/simple_overlay/g' application.json user@palette-container-id:/home/docker/sima-cli/demo/resnet_50$ sed -i 's/fakesink/simple_overlay/g' .project/pluginsInfo.json We also need to modify the plugin configuration file ``plugins/simple_overlay/overlay.json`` to include the proper caps negotiation for the input of the plugin. .. code:: javascript "caps": { "sink_pads": [ { "media_type": "application/vnd.simaai.tensor", "params": [ { "name": "format", "type": "string", "values": "DETESSDEQUANT", "json_field": null } ] } ], .. dropdown:: Step 6. Modify the ``gst`` String :animate: fade-in :color: secondary The ``gst`` string in the ``application.json`` file defines the gStreamer launch command. We need to modify this string to include the new post processing plugin. Original string: .. code:: console "gst": "simaaisrc location=/data/simaai/applications/resnet_50/etc/input_image720.rgb node-name=input delay=1 mem-target=0 index=0 loop=false ! 'video/x-raw, format=(string)RGB, width=(int)1920, height=(int)1080' ! simaaiprocesscvu name=simaai_gen_preproc_1 ! simaaiprocessmla name=simaai_process_mla_1 ! simaaiprocesscvu name=simaai_detesdequant_1 ! 'application/vnd.simaai.tensor, format=(string)DETESSDEQUANT' ! fakesink" New string: .. code:: console "gst": "simaaisrc location=/data/simaai/applications/resnet_50/etc/input_image720.rgb node-name=input delay=1 mem-target=0 index=0 loop=false ! 'video/x-raw, format=(string)RGB, width=(int)1280, height=(int)720' ! simaaiprocesscvu name=simaai_gen_preproc_1 ! simaaiprocessmla name=simaai_process_mla_1 ! simaaiprocesscvu name=simaai_detesdequant_1 ! 'application/vnd.simaai.tensor, format=(string)DETESSDEQUANT' ! simple_overlay config=/data/simaai/applications/resnet_50/etc/overlay.json" The changes in the new strings are: - ``width`` and ``height`` in the ``simaaisrc`` plugin is changed to 1280 and 720 respectively. - Added ``config`` parameter to the ``simple_overlay`` plugins. .. dropdown:: Step 7. Add Configuration File to the Custom Plugin :animate: fade-in :color: secondary In order for the compilation process to include the configuration file for the custom plugin, it must also be declared explicitly in the ``application.json`` file. Replace the ``resources`` block in the ``simple_overlay`` block in ``application.json`` as below. .. code:: javascript { ... ... ... "resources": {"configs": ["cfg/overlay.json"]} } After the changes the ``application.json`` file should look like this: .. button-link:: ../../../_static/pipeline-visualizer/app.html?data=hellosima-resnet50-palette-after-change.json :color: info :shadow: View application.json (changed) .. dropdown:: Step 8. Understand Resnet50 Output :animate: fade-in :color: secondary ``ResNet-50`` is typically trained on ImageNet (1000 classes), and its output is a 1D tensor (vector) of size 1000, representing the classification probabilities for each class. **Output Shape** .. code:: console (1, 1000) # (Batch Size, Number of Classes) Each value in this vector represents the probability score for a class. For example: .. code:: console [0.001, 0.003, ..., 0.92, ..., 0.0005] # 1000 values - The index with the highest value is the predicted class. - Example: If index 340 = 0.92, then the image is classified as zebra. The plugin ``detesdequant`` is configured to output ``NHWC`` format, which stands for: - N → Batch size (number of images processed together) - H → Height (number of rows in the image) - W → Width (number of columns in the image) - C → Channels (number of color channels, e.g., 3 for RGB or 1 for grayscale) .. dropdown:: Step 9. Add Custom Logic to Process Resnet50 Output :animate: fade-in :color: secondary Once you understand the output of the ``detesdequant`` plugin, you can write a custom plugin to process the output. Use the following code to replace the ``UserContext::run`` function in the simple_overlay/payload.cpp .. code:: c++ int argmax(float *probabilities, std::size_t size) { if (size == 0) return -1; int max_idx = 0; float max_value = probabilities[0]; for (std::size_t i = 1; i < size; i++) { if (probabilities[i] > max_value) { max_value = probabilities[i]; max_idx = i; } } return max_idx; } void UserContext::run(std::vector &input, std::span output) { // Extract the input probabilities float *probabilities = reinterpret_cast(input[0].getData().data()); std::size_t probabilities_size = input[0].getDataSize(); // Ensure valid input data if (probabilities == nullptr || probabilities_size == 0) { std::cerr << "Error: Invalid input data." << std::endl; return; } // Convert byte size to element count std::size_t num_elements = probabilities_size / sizeof(float); if (num_elements == 0) { std::cerr << "Error: No valid data in input." << std::endl; return; } // Perform the argmax operation int max_idx = argmax(probabilities, num_elements); // Print the class index and its probability std::cout << "Predicted Class Index: " << max_idx << std::endl; std::cout << "Probability: " << probabilities[max_idx] << std::endl; } The provided C++ code above processes the output of the ``detesdequant`` plugin and determines the most probable class based on its probability distribution. **argmax Function** - Iterates through an array of floating-point probabilities. - Finds the index of the highest probability value (i.e., the most confident prediction). - Returns the index of the class with the highest probability. **UserContext::run Function** - Extracts the probability tensor from the input. - Validates the input to ensure it is not empty. - Computes the most probable class using argmax(). - Prints the predicted class index and its probability. .. dropdown:: Step 10. Compile and Deploy the Project :animate: fade-in :color: secondary .. include :: ../blocks/palette_create_connect_deploy.rst To connect to a PCIe device, use ``mpk device connect -s 0`` where ``0`` stands for the slot number where the PCIe card is installed. .. dropdown:: Step 11. Check the Runtime Output :animate: fade-in :color: secondary .. note:: Connect to the DevKit console using SSH. For PCIe mode, connect through the virtual ethernet interface as described `here <../setup_pcie_mode.html#virtual-network>`_. When the pipeline is running, you can use the following command to check the running gst process's full command line. .. code:: console davinci:/var/log$ cat /proc/$(pgrep -f gst_app)/cmdline | tr '\0' ' ' /data/simaai/applications/resnet_50/bin/gst_app --manifest-json=/data/simaai/applications/resnet_50/manifest.json --gst-string=simaaisrc location=/data/simaai/applications/resnet_50/etc/input_image720.rgb node-name=input blocksize=2764800 delay=1 mem-target=0 ! 'video/x-raw, format=(string)RGB, width=(int)1280, height=(int)720' ! simaaiprocesscvu_new name=simaai_gen_preproc_1 config=/data/simaai/applications/resnet_50/etc/preproc.json ! simaaiprocessmla_new name=simaai_process_mla_1 config=/data/simaai/applications/resnet_50/etc/mla.json ! simaaiprocesscvu_new name=simaai_detesdequant_1 config=/data/simaai/applications/resnet_50/etc/detess_dequant.json ! 'application/vnd.simaai.tensor, format=(string)DETESSDEQUANT' ! simple_overlay config=/data/simaai/applications/resnet_50/etc/overlay.json davinci:/var/log$ If you want to check the process console log, you can tail ``/tmp/simaai/resnet_50_Pipeline/resnet50_mpk_Pipeline.1/gst_app.log``: .. code:: console davinci:/var/log$ tail -f /tmp/simaai/resnet_50_Pipeline/resnet50_mpk_Pipeline.1/gst_app.log Predicted Class Index: 867 Probability: 0.435257 Predicted Class Index: 867 Probability: 0.435257 Predicted Class Index: 867 Probability: 0.435257 Predicted Class Index: 867 Probability: 0.435257 Predicted Class Index: 867 Probability: 0.435257 Predicted Class Index: 867 Probability: 0.435257 Predicted Class Index: 867 Probability: 0.435257 Predicted Class Index: 867 Probability: 0.435257 Predicted Class Index: 867 Probability: 0.435257 To run the pipeline directly from the DevKit, first stop the running process, then execute the command below. This approach offers greater flexibility for debugging the pipeline without relying on the MPK process. It also demonstrates how the pipeline can be integrated into a larger application in standalone mode. For example, you can add the command to the DevKit's startup script to automatically run the pipeline at boot. .. code:: console davinci:/var/log$ sudo kill $(pgrep -f gst_app) davinci:/var/log$ sudo GST_DEBUG=3 LD_LIBRARY_PATH=/data/simaai/applications/resnet_50/lib GST_PLUGIN_PATH=/data/simaai/applications/resnet_50/lib /data/simaai/applications/resnet_50/bin/gst_app --manifest-json=/data/simaai/applications/resnet_50/manifest.json --gst-string="simaaisrc location=/data/simaai/applications/resnet_50/etc/input_image720.rgb node-name=input delay=1 mem-target=0 index=0 loop=false ! 'video/x-raw, format=(string)RGB, width=(int)1280, height=(int)720' ! simaaiprocesscvu name=simaai_gen_preproc_1 ! simaaiprocessmla name=simaai_process_mla_1 ! simaaiprocesscvu name=simaai_detesdequant_1 ! 'application/vnd.simaai.tensor, format=(string)DETESSDEQUANT' ! simple_overlay config=/data/simaai/applications/resnet_50/etc/overlay.json" The command is constructed with the following parts: - ``GST_DEBUG``: standard gstreamer debug level environment. - ``LD_LIBRARY_PATH``: path to the library. - ``GST_PLUGIN_PATH``: path to the plugin. - ``/data/simaai/applications/resnet_50/bin/gst_app``: the path to the ``gst_app`` binary. - ``--manifest-json``: the path to the ``manifest.json`` file. - ``--gst-string``: the gstreamer pipeline string. You can get this string from the manifest.json file, or using the command mentioned in the beginning of this step to get the full cmdline.