.. _gstreamer: GStreamer Example Pipelines ########################### To get started with GStreamer based Application development on the SiMa hardware, developers need to have a foundational understanding of GStreamer and in particular the ``gst-launch`` utility. The Applications are deployed on the SiMa.ai hardware as pipelines formed by a combination of open sourced GStreamer plugins and SiMa proprietary plugins and are executed with the ``gst-launch`` utility. The block diagram below captures a simplistic view of the set up any Developer would be working with for iterative development. The two main components of the setup are as follows: * **Host** * The Host machine (running the SiMa.ai supported OS) is where the developers install the latest Palette SDK and then securely connect to the SiMa.ai hardware to get started with iterative application development and eventually deploying to the board. * **SiMa.ai Hardware** * The SiMa.ai hardware refers to the boards of all form factors. .. image:: media/DeveloperSetup.png :scale: 25% :align: center :alt: Developer setup Application Overview -------------------- * Ingress - captures all supported inputs which can be streamed to the SiMa hardware for processing * Egress - captures all supported output formats which can be streamed out of the SiMa hardware * Decode/Encode - captures the multimedia decoding and encoding for the processing * Model Compute: * pre process - refers to the compute required by the model to be able to run the inference * model inference - refers to the actual inference which happens on the pre processed data * post process - refers to the compute on the data post the inference Before we dive into GStreamer development with the MPK toolchain, we will run our first Hello World application based on GStreamer directly on the board. This warm-up exercise will introduce the Developer to the following: * The (ML/AI) Application as a GStreamer pipeline with both open sourced and SiMa plugins. * Running the GStreamer pipeline using the gst-launch utility on the SiMa hardware. * Mapping the ``gst-launch`` string to bigger building blocks of an Application (Ingress, Decode, Pre-Process, Model Inference, Post-Process, Egress) All of this will help you better understand the example GStreamer pipelines shipped as part of the Palette SDK, modify them if needed and eventually build custom pipelines to be deployed on the board. Since the Hello World application will be directly run on the board, make sure you have reviewed the instructions described earlier in this documentation and are able to securely login into the SiMa hardware and access the command terminal. Running Gstreamer Pipelines --------------------------- Pre-requisites ============== The following requirements must be met before you run any application pipeline. * Palette Software is already installed on your host machine by following :ref:`Palette installation`. * GStreamer is already installed on your host machine by following `GStreamer installation `_. * You are already running the Palette CLI using ``python3 start.py`` and you are in the Docker terminal. * A device is connected to your host machine using an Ethernet or PCIe setup. Hello World Application ----------------------- Steps ===== Using the GStreamer command(s), follow the steps below. #. Run on the Host machine to stream a sample test video to the SiMa hardware. .. code-block:: console sima-user@sima-user-machine:~$ sudo gst-launch-1.0 -v videotestsrc ! videoscale ! video/x-raw,width=1280,height=720, format=NV12 ! x264enc ! rtph264pay ! queue ! udpsink host= port= #. Run on the board to implement the Hello World application. .. code-block:: console sima-user@sima-user-machine:~$ sudo gst-launch-1.0 -v udpsrc port= ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! 'video/x-h264,parsed=true,stream-format=(string)byte-stream,alignment=(string)au' ! simaaidecoder dec-width=1280 dec-height=720 dec-fps=30 ! queue ! simaaiencoder enc-width=1280 enc-height=720 enc-bitrate=7000 enc-fmt=NV12 enc-profile=high ! rtph264pay config-interval=1 pt=96 ! queue ! udpsink host= port= #. Run on the Host machine to receive the decoded stream from the board. .. code-block:: console sima-user@sima-user-machine:~$ GST_DEBUG=0 sudo gst-launch-1.0 udpsrc port=6000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! 'video/x-h264,stream-format=byte-stream,alignment=au' ! avdec_h264 ! autovideosink .. image:: media/command_breakdown.png :alt: Command breakdown Description =========== * The Application pipeline running on the board receives a sample test video pattern from the Host machine. * The udpsrc GStreamer plugin receives the input video, demuxes the input, and passes it to the decoder. * The decoder is appropriately configured with resolution and the frame rate to decode NV12 video frames from the incoming encoded video bitstream. * Then the encoder is configured as required and the decoded video frames are encoded and streamed out of the board back to the Host machine. .. image:: media/HelloWorldApp.png :scale: 25% :align: center :alt: Hello World Application Setup Next Steps ========== We ran a sample GStreamer based application as a pipeline directly on the board using the ``gst-launch`` utility. The Application introduced the Allegro multimedia codec hardware on the SiMa board and configuring the same within the gst-launch string to decode and encode multimedia data on the board. We also broke down the command running on the board and mapped them to the building blocks of an Application followed by a detailed description of the functionality. Once there is a good grasp on breaking down the example gst-launch command lines and mapping them by identifying the building blocks accurately the exercise can be extended to modify the building blocks, one at a time. For example, different ingress paths can be tried like the onboard camera on the host machine or a prepackaged video. Different egress paths can be tried as well streaming the output to a different port or to another machine. It is crucial to ensure the resolutions and formats and color spaces match the hardware specifications to avoid running into codec compatibility errors and also using the GStreamer plugins corresponding to the input or the output paths. Real-World Applications ----------------------- The workflow for building and deploying real-world end-to-end application pipelines on the SiMa hardware using the Palette SDK involves the MPK toolchain. Applications are deployed in a MPK bundle to the SiMa board and the ``application.json`` file in that package contains the gst-launch command pertaining to the application being deployed. You also get access to optimized and prepackaged pipelines in the Palette SDK which is shipped as a Docker container once you successfully walk through the steps outlined in the Quick Start Guide. We can proceed with breaking down the command line and identifying the building blocks and describing the functionality of one of the prepackaged pipelines - OpenPose Application. .. OpenPose Application .. -------------------- .. GStreamer Commands .. ================== .. .. code-block:: console .. davinci$ sudo gst-launch-1.0 -v --gst-plugin-path='/usr/local/lib/simaai/OpenPose' rtspsrc location= ! rtph264depay ! h264parse ! 'video/h264,stream-format=byte-stream' ! allegrodec2 config=\"/usr/local/etc/simaai/OpenPose/h.264_decoder_v2_1.json\" ! tee name=source ! process-cvu source-node-name=\"allegrodec\" buffers-list=\"allegrodec\" config=\"/usr/local/etc/simaai/OpenPose/cnetprocesscvupreprocess_1.json\" ! process2 config=\"/usr/local/etc/simaai/OpenPose/processv2_1.json\" ! postprocess2 config=\"/usr/local/etc/simaai/OpenPose/open_pose_postprocess_1.json\" ! groupkeypoints2 config=\"/usr/local/etc/simaai/OpenPose/group_key_points_1.json\" ! overlay. source. ! simaai-overlay2 name=overlay width=1280 height=720 format=2 renderinfo=\"input::allegrodec,pose::a65-keypoints\" render-rules=\"pose\" ! allegroenc2 config=\"/usr/local/etc/simaai/OpenPose/h.264_encoder_v2_1.json\" ! h264parse ! rtph264pay ! udpsink host= port= .. .. image:: media/command_breakdown2.png .. :alt: Command breakdown .. Description .. =========== .. * The Application pipeline running on the board receives a video input from RTSP based camera source .. * The rtspsrc GStreamer plugin receives the input video from camera and passes it to video decoder and the decoder generates NV12 video frames by decoding the input H264 stream which goes as input to Pre-process block. .. * The rtspsrc GStreamer plugin receives the input video, demuxes the input and passes it to the decoder .. * The decoder is appropriately configured with resolution and the frame rate to decode NV12 video frames from the incoming encoded video bitstream and passes it to the pre-process plugin .. * The pre-process plugin does YUV2RGB conversion, resize, quantize, normalize and tessellate, prepping the video frames for the OpenPose model .. * OpenPose model outputs key points which are grouped and sent to the overlay plugin. The Overlay plugin, overlays the key points over the input images and sends it out to the encoder. .. * The overlay-ed image is encoded and streamed back to the Host machine. .. Next Steps .. ========== .. The gst-launch string within the json file needs to be appropriately edited to match the end-to-end requirements of the Application. It is easier to make changes to the pipeline once you can accurately identify the building blocks namely Ingress, Encode, Pre-process, Model Inference, Post-process, Decode and Egress mentioned earlier. .. Once you can map the blocks you need to find the appropriate GStreamer plugins which can support the intended functionality. And to find all the available plugins on the SiMa hardware you can run the following command directly on the board, .. .. code-block:: console .. davinci$ sudo gst-inspect-1.0 .. Go to the MPK toolchain section for instructions on how to create, deploy, and visualize the output of the application pipeline we just modified. The following two MPK commands can be run on the Host machine: .. .. code-block:: console .. sima-user@docker-image-id:/home$ mpk create -s -d .. sima-user@docker-image-id:/home$ mpk deploy -f Running Application Pipelines Single Command (Beta) =================================================== .. button-link:: https://docs.sima.ai/pkg_downloads/SDK1.3.0/pipeline_runner.zip :color: primary :shadow: Download the tool **Prerequisites** You must have installed ``ffmpeg`` in your host system. Your board must be reachable from the host device through SSH. #. Install the following python libraries: .. code-block:: console sima-user@sima-user-machine:~$ pip install paramiko sima-user@sima-user-machine:~$ pip install tqdm #. Unzip and place the file ``pipeline_runner.zip`` under your workspace folder: .. code-block:: console sima-user@sima-user-machine:~$ cd ~/Downloads sima-user@sima-user-machine:~/Downloads$ unzip pipeline_runner.zip sima-user@sima-user-machine:~/Downloads$ mv pipeline_runner ~/workspace #. Run the script from within the folder: .. code-block:: console sima-user@sima-user-machine:~$ cd ~/workspace/pipeline_runner sima-user@sima-user-machine:~/workspace/pipeline_runner$ python3 pipeline_runner.py -h usage: pipeline_runner.py [-h] -hi HOST_IP_ADDRESS -b BOARD_IP_ADDRESS [-r RECEIVER_PORT] [-rstp RSTP_PORT] [-n {people-tracker,open-pose,yolov5,yolov7,efficientdet,detr}] [-d PIPELINE_DIR] [-f VIDEO_FILE] Demonstration of running pipelines automatically optional arguments: -h, --help show this help message and exit -hi HOST_IP_ADDRESS, --host-ip-address HOST_IP_ADDRESS IP Address of your host machine -b BOARD_IP_ADDRESS, --board-ip-address BOARD_IP_ADDRESS IP Address of your SiMa board -r RECEIVER_PORT, --receiver-port RECEIVER_PORT Port to receive the stream from the board -rstp RSTP_PORT, --rstp-port RSTP_PORT Port to host the rstp stream -n {people-tracker,open-pose,yolov5,yolov7,efficientdet,detr}, --pipeline-name {people-tracker,open-pose,yolov5,yolov7,efficientdet,detr} Pipeline name -d PIPELINE_DIR, --pipeline-dir PIPELINE_DIR Path to the folder where the pipelines are contained -f VIDEO_FILE, --video-file VIDEO_FILE Video file path #. Running OpenPose pipeline: .. code-block:: console sima-user@sima-user-machine:~/workspace/pipeline_runner$ python pipeline_runner.py -hi 10.42.0.1 -b 10.42.0.241 -n open-pose .. note:: There is a known issue where you have to run the script twice in order for the pipelines to run the first time you launch the script. Just re-run the script and it should show the screen with the video detections. There is also a limitation in the achieveable Frame rate due to the JIT locker of python on threads. Additionally, Detr is trained to detect cars. Therefore, the current provided video will show no detections. Manually Running Application Pipelines ====================================== In order to run the pipelines manually we will need: #. One window waiting to receive the video with the predictions from the ML pipeline. #. One deployed ML Pipeline running on the board. #. One stream sending the video to the pipeline from the host machine or a camera feed. These steps will have to be run in that specific order. Receiver Window *************** #. On your host, open a terminal or tab. Set up the receiver stream. In this example we will use ```` = ``10.42.0.1`` and ```` = ``9000``, this has to match your changes in ``application.json`` later: .. code-block:: console sima-user@sima-user-machine:~$ sudo gst-launch-1.0 udpsrc port= ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! 'video/x-h264,stream-format=byte-stream,alignment=au' ! avdec_h264 ! fpsdisplaysink sync=0 [sudo] password for sima-user: Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Pipeline Deployment ******************* #. Connect to your board using the Docker container Palette CLI. .. code-block:: console sima-user@docker-image-id:/usr/local/simaai/app_zoo/Gstreamer/PeopleTracker# mpk device connect -t sima@ ℹ Please enter the password for 10.42.0.241 🔐 : 🔗 Connection established to 10.42.0.241 . #. Choose your pipeline: .. collapse:: PeopleTracker .. code-block:: console sima-user@docker-image-id:/home# PIPELINE_PATH='/usr/local/simaai/app_zoo/Gstreamer/PeopleTracker' .. collapse:: OpenPose .. code-block:: console sima-user@docker-image-id:/home# PIPELINE_PATH='/usr/local/simaai/app_zoo/Gstreamer/OpenPose' .. collapse:: YoloV5 .. code-block:: console sima-user@docker-image-id:/home# PIPELINE_PATH='/usr/local/simaai/app_zoo/Gstreamer/YoloV5' .. collapse:: YoloV7 .. code-block:: console sima-user@docker-image-id:/home# PIPELINE_PATH='/usr/local/simaai/app_zoo/Gstreamer/YoloV7' .. collapse:: EfficientDet .. code-block:: console sima-user@docker-image-id:/home# PIPELINE_PATH='/usr/local/simaai/app_zoo/Gstreamer/EfficientDet' #. From the docker copy the file to your workspace, we will create a copy to maintain the original file: .. code-block:: sima-user@docker-image-id:/home# cp $PIPELINE_PATH/application.json $PIPELINE_PATH/application_og.json sima-user@docker-image-id:/home# cp $PIPELINE_PATH/application.json /home/docker/sima-cli/application.json #. Give ``application.json`` the right permissions to edit it: .. code-block:: console sima-user@docker-image-id:/home# chmod 666 /home/docker/sima-cli/application.json #. From your host machine use your favorite editor to edit the ``application.json`` file, I will be using VSCode. .. code-block:: console sima-user@docker-image-id:~# code ~/workspace/application.json #. Choose your input option and change the ``application.json`` accordingly: .. collapse:: USB Camera * Change ``rtspsrc location=`` for the following string ``udpsrc port= ! application/x-rtp,encoding-name=H264,payload=96`` and a port number, for example let's use ``udpsrc port=5000 ! application/x-rtp,encoding-name=H264,payload=96``. * Change ```` for a number, for example let's use ``9000``. * Change ```` for the host IP address, in my case it is ``10.42.0.1``. .. collapse:: RTSP Camera Stream * Change ```` for the camera IP address, in my case it is ``rtsp://192.168.132.205/axis-media/media.amp``. * Change ```` for a number, for example let's use ``9000``. * Change ```` for the host IP address, in my case it is ``10.42.0.1``. .. collapse:: Video file * Change ```` for the video IP address, in my case it is ``rtsp://10.42.0.1:8554/mystream``. * Change ```` for a number, for example let's use ``9000``. * Change ```` for the host IP address, in my case it is ``10.42.0.1``. Now we will have to create the stream for our video file, first we will run a docker image as a server: .. code-block:: console sima-user@sima-user-machine:~$ docker run --name rtsp_server --rm -e MTX_PROTOCOLS=tcp -p 8554:8554 aler9/rtsp-simple-server Then we will send our video stream: .. code-block:: console sima-user@sima-user-machine:~$ ffmpeg -re -nostdin -stream_loop -1 -i .mp4 -c:v copy -f rtsp rtsp://{HOST_IP_ADDRESS}:8554/mystream #. On the docker, replace the ``application.json``. .. code-block:: console sima-user@docker-image-id:/home# cp /home/docker/sima-cli/application.json $PIPELINE_PATH/ #. Create the ``.mpk`` file. .. code-block:: console sima-user@docker-image-id:/home# mpk create -s $PIPELINE_PATH -d $PIPELINE_PATH ℹ Step COMPILE completed successfully. ℹ Step COPY RESOURCE completed successfully ℹ Step RPM BUILD completed successfully. ℹ Successfully created MPK at 'Your/Chosen/Pipeline/Path/project.mpk' #. Deploy the pipeline to the board. .. code-block:: console sima-user@docker-image-id:/home# mpk deploy -f $PIPELINE_PATH/project.mpk -t 🚀 Sending MPK to 10.42.0.241... Transfer Progress for project.mpk: 100.00% 🏁 MPK sent successfully! ✔ MPK Deployment is successful for project.mpk #. Once the pipeline is deployed if you are using an IP Camera or Video File you should see the video right away. However, if you are using an USB Camera run this command on a new terminal/tab: .. code-block:: console sima-user@sima-user-machine:~$ sudo gst-launch-1.0 -vvv v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480,framerate=30/1 ! videoscale ! videoconvert ! video/x-raw, framerate=30/1, format=NV12, width=1280, height=720 ! x264enc tune=zerolatency bitrate=500 speed-preset=superfast ! rtph264pay ! udpsink host= port= Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock /GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, width=(int)640, height=(int)480, framerate=(fraction)30/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:16:1 ... Redistribute latency... 0:00:00 #. For this pipeline, the USB camera is in ``/dev/video0``, if your camera is in a different address, change the path appropriately. Running the YOLO V7 4-Camera Application Pipeline via Ethernet -------------------------------------------------------------- Steps ===== #. Install the Palette on the host machine in the Docker container. #. Launch the Docker container. #. Flash the MLSoC device with the test image (emmc). #. Connect to your board using the Docker container Palette. .. code-block:: console sima-user@docker-image-id:/home# mpk device connect -t sima@ ℹ Please enter the password for 10.42.0.241 🔐 : 🔗 Connection established to 10.42.0.241 . #. Go to the folder as shown below and download the configuration file for the YOLO V7 4-Camera pipeline. .. code-block:: console sima-user@docker-image-id:/home$ cd /usr/local/simaai/app_zoo/Gstreamer/YoloV7_4cams sima-user@docker-image-id:/usr/local/simaai/app_zoo/Gstreamer/YoloV7_4cams$ #. Edit the ``application.json`` configuration file and change the rtspsrc ports. This is the command that we are looking to change: .. code-block:: console "gst": "LD_LIBRARY_PATH='\/data\/simaai\/applications\/YoloV7_4cams\/lib' GST_DEBUG=\"0\" gst-launch-1.0 --gst-plugin-path='\/data\/simaai\/applications\/YoloV7_4cams\/lib' rtspsrc location= ! rtph264depay wait-for-keyframe=true ! h264parse ! 'video\/x-h264,parsed=true,stream-format=(string)byte-stream,alignment=(string)au' ! simaaidecoder name=simaaidecoder1 op-buff-name=simaaidecoder1 dec-width=1280 dec-height=720 dec-fmt=NV12 next-element=CVU op-buff-cached=true ! mux_1. rtspsrc location= ! rtph264depay wait-for-keyframe=true ! h264parse ! 'video\/x-h264,parsed=true,stream-format=(string)byte-stream,alignment=(string)au' ! simaaidecoder name=simaaidecoder2 op-buff-name=simaaidecoder2 dec-width=1280 dec-height=720 dec-fmt=NV12 next-element=CVU op-buff-cached=true ! mux_1. rtspsrc location= ! rtph264depay wait-for-keyframe=true ! h264parse ! 'video\/x-h264,parsed=true,stream-format=(string)byte-stream,alignment=(string)au' ! simaaidecoder name=simaaidecoder3 op-buff-name=simaaidecoder3 dec-width=1280 dec-height=720 dec-fmt=NV12 next-element=CVU op-buff-cached=true ! mux_1. rtspsrc location= ! rtph264depay wait-for-keyframe=true ! h264parse ! 'video\/x-h264,parsed=true,stream-format=(string)byte-stream,alignment=(string)au' ! simaaidecoder name=simaaidecoder4 op-buff-name=simaaidecoder4 dec-width=1280 dec-height=720 dec-fmt=NV12 next-element=CVU op-buff-cached=true ! mux_1. simaaimuxer name=mux_1 ! queue2 ! tee name=source ! queue2 ! process-cvu config=\"\/data\/simaai\/applications\/YoloV7_4cams\/etc\/yolov7_pre_proc.json\" source-node-name=\"simaaidecoder1\" buffers-list=\"simaaidecoder1,simaaidecoder2,simaaidecoder3,simaaidecoder4\" name=\"preproc\" ! process2 config=\"\/data\/simaai\/applications\/YoloV7_4cams\/etc\/yolov7_mla.json\" ! process-cvu config=\"\/data\/simaai\/applications\/YoloV7_4cams\/etc\/yolov7_post_proc.json\" source-node-name=\"mla_yolov7\" buffers-list=\"mla_yolov7\" name=\"postproc\" ! nmsyolov5 config=\"\/data\/simaai\/applications\/YoloV7_4cams\/etc\/yolov7_nms.json\" ! tee name=nms_out ! queue2 ! overlay_1. source. ! queue2 ! simaai-overlay2 name=overlay_1 width=1280 height=720 format=1 render-info=\"input::simaaidecoder1,bboxy::nmsyolov7\" render-rules=\"bboxy\" input-format=\"NV12\" labels-file=\"\/data\/simaai\/applications\/YoloV7_4cams\/share\/overlay\/labels.txt\" ! 'video\/x-raw,width=(int)1280,height=(int)720,format=(string)NV12,framerate=(fraction)30\/1' ! simaaifilter ! simaaiencoder enc-width=1280 enc-height=720 enc-bitrate=4000 name=simeaiencoder1 ! h264parse ! rtph264pay ! udpsink host= port=\"\" nms_out. ! queue2 ! overlay_2. source. ! queue2 ! simaai-overlay2 name=overlay_2 width=1280 height=720 format=1 render-info=\"input::simaaidecoder2,bboxy::nmsyolov7\" render-rules=\"bboxy\" input-format=\"NV12\" labels-file=\"\/data\/simaai\/applications\/YoloV7_4cams\/share\/overlay\/labels.txt\" ! 'video\/x-raw,width=(int)1280,height=(int)720,format=(string)NV12,framerate=(fraction)30\/1' ! simaaifilter ! simaaiencoder enc-width=1280 enc-height=720 enc-bitrate=4000 name=simaaiencoder2 ! h264parse ! rtph264pay ! udpsink host= port=\"\" nms_out. ! queue2 ! overlay_3. source. ! queue2 ! simaai-overlay2 name=overlay_3 width=1280 height=720 format=1 render-info=\"input::simaaidecoder3,bboxy::nmsyolov7\" render-rules=\"bboxy\" input-format=\"NV12\" labels-file=\"\/data\/simaai\/applications\/YoloV7_4cams\/share\/overlay\/labels.txt\" ! 'video\/x-raw,width=(int)1280,height=(int)720,format=(string)NV12,framerate=(fraction)30\/1' ! simaaifilter ! simaaiencoder enc-width=1280 enc-height=720 enc-bitrate=4000 name=simaaiencoder3 ! h264parse ! rtph264pay ! udpsink host= port=\"\" nms_out. ! queue2 ! overlay_4. source. ! queue2 ! simaai-overlay2 name=overlay_4 width=1280 height=720 format=1 render-info=\"input::simaaidecoder4,bboxy::nmsyolov7\" render-rules=\"bboxy\" input-format=\"NV12\" labels-file=\"\/data\/simaai\/applications\/YoloV7_4cams\/share\/overlay\/labels.txt\" ! 'video\/x-raw,width=(int)1280,height=(int)720,format=(string)NV12,framerate=(fraction)30\/1' ! simaaifilter ! simaaiencoder enc-width=1280 enc-height=720 enc-bitrate=4000 name=simeaiencoder4 ! h264parse ! rtph264pay ! udpsink host= port=\"\"" We will change the source streams strings from ```` to ```` for an incoming rtsp stream, it can be the same stream like in my case or a different stream. I will use ``rtsp://192.168.89.17:8554/mystream0`` for all 4. For the uspsink happening on the host, let's change ```` for the Host IP Address, in my case ``10.0.42.1`` and let's use ``8220`` for ````, ``8221`` for ````, ``8222`` for ````, ``8223`` for ````. #. Run ``mpk create``. .. code-block:: console sima-user@docker-image-id:/usr/local/simaai/app_zoo/Gstreamer/YoloV7_4cams$ mpk create -s . -d . ℹ Step a65-apps COMPILE completed successfully. ℹ Step COMPILE completed successfully. ℹ Step COPY RESOURCE completed successfully ℹ Step RPM BUILD completed successfully. ℹ Successfully created MPK at '/usr/local/simaai/app_zoo/Gstreamer/YoloV7_4cams/project.mpk' #. Run ``gst-launch`` on the host for getting an output video stream from the application on the 4 terminals with different ``UDP_PORTS`` used in the ``application.json`` file. .. code-block:: console sima-user@sima-user-machine:/home$ GST_DEBUG=0 gst-launch-1.0 udpsrc port= ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! 'video/x-h264,stream-format=byte-stream,alignment=au' ! avdec_h264 ! fpsdisplaysink sync=0 #. Deploy the mpk project on the connected device. .. code-block:: console sima-user@docker-image-id:/usr/local/simaai/app_zoo/Gstreamer/YoloV7_4cams$ mpk deploy -f project.mpk -t 🚀 Sending MPK to 192.168.135.155... Transfer Progress for project.mpk: 100.00% 🏁 MPK sent successfully! ✔ MPK Deployed! ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 ✔ MPK Deployment is successful for project.mpk. Running the YOLO V8 4-Camera Application Pipeline via Ethernet -------------------------------------------------------------- Steps ===== #. Install the Palette CLI SDK on the host machine in the Docker container. #. Launch the Docker container. #. Flash the MLSoC device with the test image (emmc). #. Connect to the MLSoC Developer Board from the Palette CLI SDK. .. code-block:: console sima-user@docker-image-id:/home$ mpk device connect -t sima@ ℹ Please enter the password for 🔐 : 🔗 Connection established to . #. Go to the folder as shown below and download the configuration file for the YOLO V8 4-Camera pipeline. .. code-block:: console sima-user@docker-image-id:/home$ cd /usr/local/simaai/app_zoo/Gstreamer/YoloV8_4cams #. Edit the ``application.json`` configuration file and change ``rtspsrc``, ``uspsink`` host IP and port in the GST command. In particular the pipeline uses 4 RTSP sources as input and generate 4 independent UDP output streams. Find out RTSP URL for the 4 IP cameras and replace , , & in the pipeline with respective Camera URLs. for example RTSP URL for Hikvision cameras looks something like ``rtsp://:@:/Streaming/channels/``. Similarly, to send output stream to host present on the same network, IP address of the host and a port number is required per stream. .. code-block:: console sima-user@docker-image-id:/home$ "gst" : "LD_LIBRARY_PATH=\/data\/simaai\/applications\/YoloV8_4cams\/lib GST_DEBUG=3 gst-launch-1.0 --gst-plugin-path=\"\/data\/simaai\/applications\/YoloV8_4cams\/lib\" rtspsrc location= ! rtph264depay wait-for-keyframe=true ! h264parse ! 'video\/x-h264,parsed=true,stream-format=(string)byte-stream,alignment=(string)au' ! simaaidecoder name=simaaidecoder1 op-buff-name=simaaidecoder1 dec-width=1280 dec-height=720 dec-fmt=NV12 next-element=CVU op-buff-cached=true ! mux_1. rtspsrc location= ! rtph264depay wait-for-keyframe=true ! h264parse ! 'video\/x-h264,parsed=true,stream-format=(string)byte-stream,alignment=(string)au' ! simaaidecoder name=simaaidecoder2 op-buff-name=simaaidecoder2 dec-width=1280 dec-height=720 dec-fmt=NV12 next-element=CVU op-buff-cached=true ! mux_1. rtspsrc location= ! rtph264depay wait-for-keyframe=true ! h264parse ! 'video\/x-h264,parsed=true,stream-format=(string)byte-stream,alignment=(string)au' ! simaaidecoder name=simaaidecoder3 op-buff-name=simaaidecoder3 dec-width=1280 dec-height=720 dec-fmt=NV12 next-element=CVU op-buff-cached=true ! mux_1. rtspsrc location= ! rtph264depay wait-for-keyframe=true ! h264parse ! 'video\/x-h264,parsed=true,stream-format=(string)byte-stream,alignment=(string)au' ! simaaidecoder name=simaaidecoder4 op-buff-name=simaaidecoder4 dec-width=1280 dec-height=720 dec-fmt=NV12 next-element=CVU op-buff-cached=true ! mux_1. simaaimuxer name=mux_1 ! queue2 ! tee name=source ! queue2 ! process-cvu config=\"\/data\/simaai\/applications\/YoloV8_4cams\/etc\/preproc.json\" source-node-name=\"simaaidecoder1\" buffers-list=\"simaaidecoder1,simaaidecoder2,simaaidecoder3,simaaidecoder4\" name=\"preproc\" ! process2 config=\"\/data\/simaai\/applications\/YoloV8_4cams\/etc\/yolov8_mla.json\" ! process-cvu config=\"\/data\/simaai\/applications\/YoloV8_4cams\/etc\/postproc.json\" source-node-name=\"yolov8_MLA\" buffers-list=\"yolov8_MLA\" name=\"postproc\" ! yolov8_box_composer config=\"\/data\/simaai\/applications\/YoloV8_4cams\/etc\/box_composer.json\" name=\"box_composer\" ! tee name=nms_out ! queue2 ! overlay_1. source. ! queue2 ! simaai-overlay2 name=overlay_1 width=1280 height=720 format=1 render-info=\"input::simaaidecoder1,bbox::yolov8_box_composer\" render-rules=\"bbox\" input-format=\"NV12\" labels-file=\"\/data\/simaai\/applications\/YoloV8_4cams\/etc\/yolov8_labels\" ! 'video\/x-raw,width=(int)1280,height=(int)720,format=(string)NV12,framerate=(fraction)30\/1' ! simaaifilter ! simaaiencoder enc-width=1280 enc-height=720 enc-bitrate=4000 name=simeaiencoder1 ! h264parse ! rtph264pay ! udpsink host= port= nms_out. ! queue2 ! overlay_2. source. ! queue2 ! simaai-overlay2 name=overlay_2 width=1280 height=720 format=1 render-info=\"input::simaaidecoder2,bbox::yolov8_box_composer\" render-rules=\"bbox\" input-format=\"NV12\" labels-file=\"\/data\/simaai\/applications\/YoloV8_4cams\/etc\/yolov8_labels\" ! 'video\/x-raw,width=(int)1280,height=(int)720,format=(string)NV12,framerate=(fraction)30\/1' ! simaaifilter ! simaaiencoder enc-width=1280 enc-height=720 enc-bitrate=4000 name=simaaiencoder2 ! h264parse ! rtph264pay ! udpsink host= port= nms_out. ! queue2 ! overlay_3. source. ! queue2 ! simaai-overlay2 name=overlay_3 width=1280 height=720 format=1 render-info=\"input::simaaidecoder3,bbox::yolov8_box_composer\" render-rules=\"bbox\" input-format=\"NV12\" labels-file=\"\/data\/simaai\/applications\/YoloV8_4cams\/etc\/yolov8_labels\" ! 'video\/x-raw,width=(int)1280,height=(int)720,format=(string)NV12,framerate=(fraction)30\/1' ! simaaifilter ! simaaiencoder enc-width=1280 enc-height=720 enc-bitrate=4000 name=simaaiencoder3 ! h264parse ! rtph264pay ! udpsink host= port= nms_out. ! queue2 ! overlay_4. source. ! queue2 ! simaai-overlay2 name=overlay_4 width=1280 height=720 format=1 render-info=\"input::simaaidecoder4,bbox::yolov8_box_composer\" render-rules=\"bbox\" input-format=\"NV12\" labels-file=\"\/data\/simaai\/applications\/YoloV8_4cams\/etc\/yolov8_labels\" ! 'video\/x-raw,width=(int)1280,height=(int)720,format=(string)NV12,framerate=(fraction)30\/1' ! simaaifilter ! simaaiencoder enc-width=1280 enc-height=720 enc-bitrate=4000 name=simeaiencoder4 ! h264parse ! rtph264pay ! udpsink host= port=" " #. Run ``mpk create``. .. code-block:: console sima-user@docker-image-id:/usr/local/simaai/app_zoo/Gstreamer/YoloV8_4cams$ mpk create -s . -d . ℹ Step a65-apps COMPILE completed successfully. ℹ Step COMPILE completed successfully. ℹ Step COPY RESOURCE completed successfully ℹ Step RPM BUILD completed successfully. ℹ Successfully created MPK at '/usr/local/simaai/app_zoo/Gstreamer/YoloV8_4cams/project.mpk' #. Run ``gst-launch`` on the host for getting an output video stream from the application on the 4 terminals with different PORTS used in the ``application.json`` file. For playing all the four streams, run below command with 4 different port number matching to ``udpsink`` port number provided on the board so that ``udpsrc`` plugin can listen to the specific ports. .. code-block:: console sima-user@sima-user-machine:/home$ GST_DEBUG=0 gst-launch-1.0 udpsrc port= ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! 'video/x-h264,stream-format=byte-stream,alignment=au' ! avdec_h264 ! fpsdisplaysink sync=1 #. Deploy the mpk project on the connected device. .. code-block:: console sima-user@docker-image-id:/usr/local/simaai/app_zoo/Gstreamer/YoloV8_4cams$ mpk deploy -f project.mpk -t 🚀 Sending MPK to ... Transfer Progress for project.mpk: 100.00% 🏁 MPK sent successfully! ✔ MPK Deployed! ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 ✔ MPK Deployment is successful for project.mpk. Generic Aggregator Plugin ------------------------- The generic Aggregator is a plugin template and not a complete plugin. This template is a tool for creating custom plugins for GStreamer application pipelines. Prerequisites ============= The following requirements must be met before you run any application pipeline. * The Palette software is installed on your host machine by following :ref:`Palette installation`. * GStreamer is installed on your host machine by following `GStreamer installation `_. * The Palette CLI is running on your host machine using ``python3 start.py`` and you are in the Docker terminal. * A device is connected to your host machine either over Ethernet or PCIe. Steps ===== #. Launch the Docker container. #. Flash the MLSoC device with the test image (emmc). #. Connect to the MLSoC Developer Board from the Palette CLI SDK. .. code-block:: console sima-user@docker-image-id:/home$ mpk device connect -t sima@ 🔗 Connection established to 192.168.135.218 . #. Follow the path shown below and download the Generic Aggregator Demo. .. code-block:: console sima-user@docker-image-id:/home$ cd /usr/local/simaai/app_zoo/Gstreamer/GenericAggregatorDemo #. Edit the json file ``plugins/simple_overlay/cfg/overlay.json`` and change ``"dump_data": 0 to "dump_data": 1``. .. code-block:: console {"version": 0.1, "node_name": "overlay", "memory": { "cpu": 0, "next_cpu": 0 }, "system": { "out_buf_queue": 1, "debug": 0, "dump_data": 0 }, "buffers": { "input": [ { "name": "image", "size": 894999 }, { "name": "boxes", "size": 32 } ], "output": { "size": 211527 } }} #. Run the ``mpk create`` command. .. code-block:: console sima-user@docker-image-id:/usr/local/simaai/app_zoo/Gstreamer/GenericAggregatorDemo$ mpk create -s . -d . ℹ Step a65-apps COMPILE completed successfully. ℹ Step COMPILE completed successfully. ℹ Step COPY RESOURCE completed successfully ℹ Step RPM BUILD completed successfully. ✔ Successfully created MPK at '/usr/local/simaai/app_zoo/Gstreamer/GenericAggregatorDemo/project.mpk' #. Deploy the mpk on the connected device. .. code-block:: console sima-user@docker-image-id:/usr/local/simaai/app_zoo/Gstreamer/GenericAggregatorDemo$ mpk deploy -f project.mpk ℹ killing older mpk with the same name deployed on 192.168.135.218 ✔ MPK with id: efficientdet_Pipeline is successfully killed on 192.168.135.218 💀 🚀 Sending MPK to 192.168.135.218... Transfer Progress for project.mpk: 100.00% 🏁 MPK sent successfully! ✔ MPK Deployed! ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 ✔ MPK Deployment is successful for project.mpk. The ``"dump_data": 1`` flag dumps the output images under ``/tmp/`` with name overlay-.out. #. Pull any frame from the device and view it in any image viewer application on the host machine. #. Observe the output image for a pair of bounding boxes drawn on an image as shown below. .. image:: media/aggregator_demo.png :alt: Generic Aggregator Demo .. note:: This is a demo application therefore the location of the bounding boxes has no meaning. They are drawn at a static location. Conclusion ---------- We started with an overview of the application pipeline and the Developer setup then moving onto briefly understanding the Palette SDK and the associated MPK and the two Development paths possible. Within GStreamer development flow we first ran a simplistic GStreamer-based pipeline directly on the board while understanding the individual components of the command line and the associated plugins. We then repeated the same exercise with an actual real-world application prepackaged within the SDK and concluded with how it can be created, deployed, and modified if and when needed. Modifying the ingress and the egress paths within an application pipeline is a good skillset to develop and once complex real-world GStreamer pipelines are included in SiMa's Developer Journey document, breaking them down into simpler building blocks and eventually changing them and deploying them becomes easier. Building custom pipelines from scratch requires the ability to re-configure/update individual GStreamer plugins which will be covered in a future version of this document.