.. _developing_gstreamer_app:

Developing & Debugging GStreamer Applications
#############################################

Introduction
============

This comprehensive guide walks developers through creating new end-to-end performant applications on MLSoC.

This guide is composed of:

- Understanding the recommended development flow
- Understanding the hardware and software components of SiMa.ai's MLSoC
- understanding the details and SW components of each of the MLSoC IPs
- understanding how to port an application by taking the following 3 steps:
  
    - **Host CPU**: Starting with an x86/mac python classification application example
        - This will serve as a reference application during our porting and development.
    - **MLSoC CPU**: Porting the x86/mac application to run fully on MLSoC Arm A65 Processors
        - Illustration of how to run any python application on MLSoC.
    - **MLSoC HW-Accelerated (Gstreamer)**: Porting the x86/mac application to run *fully optimized* on MLSoC with hardware-accelerated blocks.

Recommended development flow
============================

The high level steps of building an ML application pipeline for SiMa.ai's MLSoC is as follows:

.. image:: media/building_pipeline_high_level_flow.jpg
    :align: center
    :scale: 55%

|

.. tip::
    
    It is good practice for developers to write a reference application before porting to the MLSoC. In this guide, our reference application is a Python script running onnxruntime on CPU.

#. Have a trained model that will be used to build the application. That model will be compiled to run on the MLSoC Machine Learning Accelerator (MLA).
#. Developer writes a python script using the :ref:`modelsdk` APIs to import, quantize, test and compile the trained model (``tar.gz``). 
#. Developer extracts the contents of the ``tar.gz`` which contains the compiled model (``model.lm``) and the configuration meta-data for the model (``model_mpk.json``).
    
    * These files will be used when developing the application pipeline to configure the runtime.
#. Developer constructs the GStreamer application/pipeline using compiled model resources and writing configuration files for various SiMa.ai Gstreamer plugins. (*optional*)
#. Developer tests/debugs application using ``gst-launch`` on the board (*optional*)
#. Developer creates an ML Pipeline Package (``mpk``) project to create a application project so that the developed pipeline can be packaged and deployed in production.

In the following sections, we will go through the steps above to develop the example ResNet50 application.

.. note:: 

    This guide utilizes pre-trained models and transforms from the `torchvision` library.
    Throughout this guide we use Open Images dataset for calibrating the model.

    * For more information, visit the `Open Images official website <https://storage.googleapis.com/openimages/web/index.html>`_.
    * For more information, visit the `torchvision official documentation <https://pytorch.org/vision/stable/index.html>`_.
    * For more information, visit the `ImageNet official website <http://www.image-net.org/>`_.