Developing End-to-End Applications on MLSoC

This guide aims to assist developers in building applications on an MLSoC (System on Chip) by leveraging its specialized hardware capabilities. It provides a structured approach to porting applications from traditional x86/Mac architectures to SiMa.ai’s MLSoC environment, debugging and testing with GStreamer for high performance and efficiency.

This guide focuses on porting a classification application to run optimized for MLSoC. Advanced sections will focus on how to communicate with external systems, camera inputs, video decode/encode, etc.

What you will learn

  • How to integrate a model compiled for SiMa.ai’s MLA into an application and run inference

  • What graphs and capabilities are available for the EV74 DSP to be used with pre and post-processing

  • What hardware-accelerated APIs, libraries and plugins are available to build an application with the MLSoC

  • How to develop end-to-end pipelines using Python and Gstreamer on the MLSoC

Prerequisites

Requirement Type

Requirement Details

Palette and MLSOC Firmware

  • Palette: 1.4.0_master_B122

  • MLSoC Firmware: 1.4.0_master_B1230

Development environment setup

  • On your host development machine, follow the instructions to ensure Palette is fully installed by following the Palette Software Installation guide.

  • Have an MLSoC device or development kit available with the latest firmware installed as specified in the prerequisites.

    • Have ssh access to the development kit set up.

Development journey

When developing applications, it is recommended to follow to main steps:

  1. Develop a GStreamer pipeline using gst-launch to debug the application and build it step by step.

  2. Once the application is working as expect it, create an MPK project to package it as an mpk and be able to deploy to any MLSoC device.

Contents: