Run Demos

Before building your first pipeline, try one of the preconfigured demos. SiMa provides three distinct demo experiences:

Edgematic 🔥 is a state-of-the-art, web-based development platform for Edge AI applications. You can quickly explore the demo by dragging and dropping a prebuilt pipeline with just a few clicks.
Run a demo pipeline on the DevKit using the Palette, with a companion Ubuntu machine to supply video input and render YoloV7 results accelerated by the MLA.
Run a Large Language Model demo on the Modalix platform using sima-cli. Experience cutting-edge multimodal AI running on SiMa’s high performance power efficient Modalix platform.

Follow the instructions below to explore our demos. This also serves as a great way to verify that your environment is set up correctly.

Note

Currently Edgematic is in beta and available for free on the Amazon Web Services (AWS) Marketplace. To obtain access, click link. To sign up, you will need an AWS account.

Log into Edgematic, create a new project called demo, on the right hand side Catalog tab, find yolo_v5_ethernet application under SiMa and drag it into the Canvas. Then hit the play button the top right corner of the page.

If you have access to a DevKit, then running this demo helps you understand the general workflow of Vision AI. The following diagram illustrates the demo setup. An Ubuntu machine connects to the DevKit over the network and is configured for RTSP (Real-Time Streaming Protocol) video streaming to and from the DevKit, which handles accelerated ML tasks.

Additionally, you need to install Palette on the host machine to build and deploy the app. Palette can be installed on the same machine running the video streaming processes.

Note

This is a demo for the standalone mode. For more information on how to setup network in the standalone mode, refer to this page.

Step 4. Edit the Demo App GStreamer Pipeline

You need to edit the sample app configuration file application.json located inside the Palette container. You can use either vim or nano to edit the file.

user@palette-container-id:~$ cd /usr/local/simaai/app_zoo/Gstreamer/YoloV7
user@palette-container-id:/usr/local/simaai/app_zoo/Gstreamer/YoloV7$ vim application.json

The gst element in this file defines the gStreamer launch command, inside this command there are 3 variables need to be replaced:

<RTSP_SRC>: Replace to rtsp://<ubuntu_host_ip>:8554/mystream
<HOST_IP>: Replace to the Ubuntu host IP address
<PORT> : Replace with the Ubuntu host port where the third terminal from the previous step indicates the listening port for the video rendering process.

Click to view the explainations of the gst launch command in the application.json file

The gst launch command is used to construct and execute multimedia processing and machine learning pipeline executed in SiMa MLA. It enables real-time streaming, decoding, processing, and rendering of data streams using various GStreamer plugins. Common use cases include playing media files, capturing video from a camera, streaming RTSP sources, and applying machine learning inference on video frames. By chaining different elements, gst launch allows developers to create flexible and efficient multimedia applications.

"gst": "rtspsrc location=rtsp://<HOST_IP>:8554/mystream !\
        rtph264depay wait-for-keyframe=true ! h264parse ! 'video/x-h264, parsed=true, stream-format=(string)byte-stream, alignment=(string)au, width=(int)[1,4096], height=(int)[1,4096]' !\
        simaaidecoder sima-allocator-type=2 name='decoder' next-element='CVU' !\
        tee name=source ! 'video/x-raw' !\
        simaaiprocesscvu_new name=simaai_preprocess num-buffers=5 !\
        simaaiprocessmla_new name=simaai_process_mla num-buffers=5 !\
        simaaiprocesscvu_new name=simaai_postprocess num-buffers=5 !\
        nmsyolov5_new name=simaai_nms_yolov5 orig-img-width=1280 orig-img-height=720 ! \
        overlay. source. ! 'video/x-raw' ! \
        simaai-overlay2_new name=overlay render-info='input::decoder,bboxy::simaai_nms_yolov5' labels-file='/data/simaai/applications/YoloV7/share/overlay_new/labels.txt' !\
        simaaiencoder enc-bitrate=4000 name=encoder !\
        h264parse !\
        rtph264pay !\
        udpsink host=<HOST_IP> port=PORT"

RTSP Source:

rtspsrc location=rtsp://<HOST_IP>:8554/mystream: Retrieves the RTSP video stream from the Ubuntu host.

H.264 Stream Handling:

rtph264depay wait-for-keyframe=true: Depayloads the H.264 stream while ensuring it starts with a keyframe.
h264parse: Parses the H.264 stream.
'video/x-h264, parsed=true, stream-format=(string)byte-stream, alignment=(string)au, width=(int)[1,4096], height=(int)[1,4096]': Specifies video format constraints.

Decoding:

simaaidecoder sima-allocator-type=2 name='decoder' next-element='CVU': Uses SiMa AI’s hardware decoder and allocates memory.

Processing & Object Detection:

tee name=source: Duplicates the video stream for parallel processing.
simaaiprocesscvu_new name=simaai_preprocess num-buffers=5: Prepares frames for processing.
simaaiprocessmla_new name=simaai_process_mla num-buffers=5: Runs inference using the Machine Learning Accelerator (MLA).
simaaiprocesscvu_new name=simaai_postprocess num-buffers=5: Post-processes the model output.
nmsyolov5_new name=simaai_nms_yolov5 orig-img-width=1280 orig-img-height=720: Applies Non-Maximum Suppression (NMS) for YOLOv5 object detection.

Overlaying Bounding Boxes:

overlay. source. ! 'video/x-raw': Merges the original and processed video streams.
simaai-overlay2_new name=overlay render-info='input::decoder,bboxy::simaai_nms_yolov5' labels-file='/data/simaai/applications/YoloV7_Eth/share/overlay_new/labels.txt':
- Overlays detected objects on the video.
- Uses the provided labels file for object classification.

Encoding & Transmission:

simaaiencoder enc-bitrate=4000 name=encoder: Encodes the processed video stream.
h264parse: Parses the encoded H.264 stream.
rtph264pay: Packs the stream into RTP format.
udpsink host=<HOST_IP> port=<PORT>: Sends the processed video stream over UDP to the specified host.

This demo, nicknamed llima, integrates live image capture, voice input, and text input to enable seamless multimodal interaction powered by the LLaVA model. It also includes support for Retrieval-Augmented Generation (RAG) functionality.

Note

This demo is compatible only with the Modalix DevKit. To install the demo, ensure the DevKit is connected to the Internet. Once installed, it can run entirely offline. To install this demo on Modalix Early Access Kit, refer to this page.

Note

Please install or upgrade sima-cli before continuing. This guide is intended for use with the latest sima-cli version.