Develop with GStreamer

GStreamer is a powerful, pipeline-based framework that connects a wide range of media processing components to create complex multimedia workflows.

For example, it can be used to build a system that reads media files in one format, processes them, and outputs them in another. Thanks to its modular design, formats and processing steps can be easily swapped or extended in a plug-and-play manner.

Key Components

  • Pipelines: how GStreamer connects different media processing elements.

  • Elements: basic building blocks, for example, sources, filters, and sinks.

  • Pads: how elements connect.

  • Bins: logical groupings of elements.

  • Plugins (optional): exendable functionality.

Plugins

For a list of SiMa-specific plugins, see GStreamer Plugins.

Development Workflow

The high-level steps of building an ML application pipeline, using the GStreamer framework, for SiMa.ai’s MLSoC is listed below. It is good practice for developers to write a reference application before porting to the MLSoC. In this guide, our reference application is a Python script running onnxruntime on CPU. For detailed step-by-step instructions, see Build with Palette.

  1. Acquire a trained model that will be used to build the application. That model will be compiled to run on the MLSoC Machine Learning Accelerator (MLA).

  2. Write a Python script using the ModelSDK APIs to import, quantize, test and compile the trained model (tar.gz).

  3. Extract the contents of the tar.gz which contains the compiled model (model.lm) and the configuration meta-data for the model (model_mpk.json). These files will be used when developing the application pipeline to configure the runtime.

  4. Construct the GStreamer application/pipeline using compiled model resources and writing configuration files for various SiMa.ai GStreamer plugins (optional).

  5. Test/debug the application using gst-launch on the board (optional).

  6. Create an ML Pipeline Package (mpk) project to create an application project so that the developed pipeline can be packaged and deployed in production.

Key Features

The Palette SDK integrates GStreamer to provide a robust, high-performance multimedia framework tailored for video, audio, and imaging applications on SiMA MLSoC devices. Leveraging GStreamer’s pipeline-based architecture, the SDK enables the following features: