Reference Classification Application

In order to build an application on the MLSoC, we will first write a python application that will serve as a reference, and then begin to build our GStreamer application. In this first section, we have the code and resources for the python based application.

Resources and setup

Please download and place this archive in your Palette docker container.

Download Reference App

In your environment, uncompress the application, go into the uncompressed folder, make sure to install the requirements:

sima-user@docker-image-id$ tar xvf building_applications_mlsoc_1.4.0.tar.xz
    building_applications_mlsoc_1.4.0/
    building_applications_mlsoc_1.4.0/requirements.txt
    building_applications_mlsoc_1.4.0/data/
    building_applications_mlsoc_1.4.0/data/imagenet_labels.txt
    building_applications_mlsoc_1.4.0/data/openimages_v7_images_and_labels.pkl
    building_applications_mlsoc_1.4.0/data/golden_retriever_207.jpg
    building_applications_mlsoc_1.4.0/README.md
    building_applications_mlsoc_1.4.0/models/
    building_applications_mlsoc_1.4.0/models/download_resnet50.py
    building_applications_mlsoc_1.4.0/src/
    building_applications_mlsoc_1.4.0/src/x86_reference_app/
    building_applications_mlsoc_1.4.0/src/x86_reference_app/resnet50_reference_classification_app.py
    building_applications_mlsoc_1.4.0/src/modelsdk_quantize_model/
    building_applications_mlsoc_1.4.0/src/modelsdk_quantize_model/resnet50_quant.py
sima-user@docker-image-id$ cd building_applications_mlsoc_1.4.0/
sima-user@docker-image-id$ tree -L 3
    .
    ├── data
    │   ├── golden_retriever_207.jpg
    │   ├── imagenet_labels.txt
    │   └── openimages_v7_images_and_labels.pkl
    ├── models
    │   └── download_resnet50.py
    ├── README.md
    ├── requirements.txt
    └── src
        ├── modelsdk_quantize_model
        │   └── resnet50_quant.py
        └── x86_reference_app
            └── resnet50_reference_classification_app.py
sima-user@docker-image-id$ pip3 install -r requirements.txt

Note

  • data: Any data we will use in the reference application or the quantization script

    • golden_retriever_207.jpg: A test image we will use in our scripts.

    • imagenet_labels.txt: The list of labels from ImageNet we will use to perform inference and retrieve what class was predicted.

    • openimages_v7_images_and_labels.pkl: The labels and images from Open Images that we will use to calibrate our model during quantization.

  • models: Where we will store our model

    • download_resnet50.py: Downloads a pre-trained ResNet50 model, add a softmax node at the end, and saves the model.

    • The output of this script is the core model we will be using throughout the guide.

  • src: Application and quantization scripts
    • modelsdk_quantize_model/resnet50_quant.py:

      • Script used to quantize our resnet50_model.onnx model using SiMa.ai’s Palette ModelSDK APIs (output from download_resnet50.py).

    • x86_reference_app/resnet50_reference_classification_app.py:

      • Our sample python application that can run on any machine and uses onnxruntime to run the resnet50_model.onnx along with all pre and post processing.

      • his will be our reference application, and we will be developing this exact application on the MLSoC.

Python example for x86/Mac

Download and save the model

Throughout this guide, we will be using a ResNet50 model from TorchVision and we will add a softmax to the output. To download the model and add the softmax operator, the output will be stored in the directory models/resnet50_model.onnx:

sima-user@docker-image-id$ python models/download_resnet50.py
    Downloading: "https://download.pytorch.org/models/resnet50-11ad3fa6.pth" to /root/.cache/torch/hub/checkpoints/resnet50-11ad3fa6.pth
    100%|███████████████████████████████████████████████████████████████████████████████████████████| 97.8M/97.8M [00:00<00:00, 114MB/s]
    ============== Diagnostic Run torch.onnx.export version 2.0.1+cpu ==============
    verbose: False, log level: Level.ERROR
    ======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

    Model exported successfully to /home/docker/sima-cli/building_applications_mlsoc_1.4.0/models/resnet50_export.onnx
    Simplified model saved to /home/docker/sima-cli/building_applications_mlsoc_1.4.0/models/resnet50_model.onnx
console$ tree -L3
    .
    ├── data
    │   ├── golden_retriever_207.jpg
    │   ├── imagenet_labels.txt
    │   └── openimages_v7_images_and_labels.pkl
    ├── models
    │   ├── download_resnet50.py
    │   └── resnet50_model.onnx
    ├── README.md
    ├── requirements.txt
    └── src
        ├── modelsdk_quantize_model
        │   └── resnet50_quant.py
        └── x86_reference_app
            └── resnet50_reference_classification_app.py

Run the reference application

The following Python script demonstrates how to classify images using an ONNX ResNet50 model and onnxruntime. This script can be run on any x86/Mac machine with the necessary dependencies installed.

The reference application itself:

resnet50_reference_classification_app.py
  1#**************************************************************************
  2#||                        SiMa.ai CONFIDENTIAL                          ||
  3#||   Unpublished Copyright (c) 2023-2024 SiMa.ai, All Rights Reserved.  ||
  4#**************************************************************************
  5# NOTICE:  All information contained herein is, and remains the property of
  6# SiMa.ai. The intellectual and technical concepts contained herein are
  7# proprietary to SiMa and may be covered by U.S. and Foreign Patents,
  8# patents in process, and are protected by trade secret or copyright law.
  9#
 10# Dissemination of this information or reproduction of this material is
 11# strictly forbidden unless prior written permission is obtained from
 12# SiMa.ai.  Access to the source code contained herein is hereby forbidden
 13# to anyone except current SiMa.ai employees, managers or contractors who
 14# have executed Confidentiality and Non-disclosure agreements explicitly
 15# covering such access.
 16#
 17# The copyright notice above does not evidence any actual or intended
 18# publication or disclosure  of  this source code, which includes information
 19# that is confidential and/or proprietary, and is a trade secret, of SiMa.ai.
 20#
 21# ANY REPRODUCTION, MODIFICATION, DISTRIBUTION, PUBLIC PERFORMANCE, OR PUBLIC
 22# DISPLAY OF OR THROUGH USE OF THIS SOURCE CODE WITHOUT THE EXPRESS WRITTEN
 23# CONSENT OF SiMa.ai IS STRICTLY PROHIBITED, AND IN VIOLATION OF APPLICABLE
 24# LAWS AND INTERNATIONAL TREATIES. THE RECEIPT OR POSSESSION OF THIS SOURCE
 25# CODE AND/OR RELATED INFORMATION DOES NOT CONVEY OR IMPLY ANY RIGHTS TO
 26# REPRODUCE, DISCLOSE OR DISTRIBUTE ITS CONTENTS, OR TO MANUFACTURE, USE, OR
 27# SELL ANYTHING THAT IT  MAY DESCRIBE, IN WHOLE OR IN PART.
 28#
 29#**************************************************************************
 30
 31import cv2
 32import shutil
 33import argparse
 34import numpy as np
 35import onnxruntime as ort
 36
 37from pathlib import Path
 38from typing import Union
 39
 40# Constants
 41ROOT_PATH = Path(__file__).parent.resolve()
 42INPUT_IMAGES_PATH = str(Path(ROOT_PATH, "../../data/"))
 43IMAGENET_LABELS_PATH = str(Path(ROOT_PATH, "../../data/imagenet_labels.txt"))
 44MODEL_PATH = str(Path(ROOT_PATH, "../../models/resnet50_model.onnx"))
 45SAVE_OUTPUTS_FOR_DEBUG = True
 46OUTPUT_DEBUG_PATH = ROOT_PATH/"debug"
 47
 48####################
 49# Helper Functions #
 50####################
 51
 52def recreate_directory(dir_path: Union[Path, str]):
 53    # Ensure dir_path is a Path object if given a string
 54    dir_path = Path(dir_path)
 55
 56    # Check if the directory exists
 57    if dir_path.exists():
 58        # Remove the directory and all its contents
 59        shutil.rmtree(dir_path)
 60        print(f"Deleted directory: {dir_path}")
 61
 62    # Recreate the directory
 63    dir_path.mkdir(parents=True, exist_ok=True)
 64    print(f"Recreated directory: {dir_path}")
 65
 66###################
 67# Setup Functions #
 68###################
 69
 70def setup(model_path: str, labels_path: str):
 71    # Load labels and ONNX model
 72    with open(labels_path, "r") as f:
 73        labels = [line.strip() for line in f.readlines()]
 74
 75    # Load the ONNX model
 76    session = ort.InferenceSession(model_path)
 77    input_name = session.get_inputs()[0].name
 78
 79    return labels, session, input_name
 80
 81######################
 82# Pipeline Functions #
 83######################
 84
 85def read_image(image_path: str, dump_output: bool = False):
 86    image = cv2.imread(str(image_path), cv2.IMREAD_UNCHANGED)
 87    rgb_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
 88
 89    image_name = Path(image_path).stem
 90    if dump_output:
 91        debug_image_name = image_name + "_rgb.bin"
 92        rgb_image.tofile(str(OUTPUT_DEBUG_PATH/debug_image_name))
 93
 94    return rgb_image, image_name
 95
 96
 97# Function to preprocess the image
 98def preprocess_image(image: np.ndarray, input_shape: tuple = (224, 224), scale_factor: tuple = 255.0,
 99                    dump_output_image_name: str = None, dump_output: bool = False):
100    mean = [0.485, 0.456, 0.406]
101    stddv = [0.229, 0.224, 0.225]
102
103    resized_image = cv2.resize(image, input_shape)
104    image_data = resized_image / scale_factor
105    image_data = (image_data - mean) / stddv
106
107    if dump_output:
108        # Dump reference output before transposing because MLA uses NHWC format not NCHW
109        assert dump_output_image_name is not None
110
111        debug_image_path = OUTPUT_DEBUG_PATH/f"{dump_output_image_name}_preprocessed_rgb_nhwc_fp32.bin"
112        image_data.tofile(debug_image_path)
113
114    image_data = np.transpose(image_data.astype('float32'), (2, 0, 1))  # Change to (C, H, W)
115    image_data = np.expand_dims(image_data, axis=0)  # Add batch dimension
116
117    return image_data
118
119# Function to post-process the output
120def postprocess_output(output: np.ndarray, labels: dict,
121                    dump_output_image_name: str = None, dump_output: bool = False):
122    probabilities = output[0][0]
123    max_idx = np.argmax(probabilities)
124
125    if dump_output:
126        # Dump reference output before transposing because MLA uses NHWC format not NCHW
127        assert dump_output_image_name is not None
128
129        debug_image_path = OUTPUT_DEBUG_PATH/f"{dump_output_image_name}_inference_output_probabilities.bin"
130        probabilities.tofile(debug_image_path)
131
132    return labels[max_idx], probabilities[max_idx]
133
134def main(images_path: str, model_path: str, labels_path: str = IMAGENET_LABELS_PATH):
135    """ Runs an application pipeline composed of the following 5 stages:
136
137        Load image -> Preprocess image -> Run ResNet50 (inference) -> Postprocess outputs -> Display results
138    """
139
140    # Initialization #
141    labels, inference_session, input_name = setup(model_path=model_path, labels_path=labels_path)
142    image_paths = list(Path(images_path).glob("*.jpg"))
143
144    # 5 Stage application pipeline #
145    print(f"\nProcessing all images in: {images_path}\n")
146
147    for idx, image_path in enumerate(image_paths):
148        print(f"Processing image [{idx}] --> ", end="")
149
150        # Load image #
151        image, image_name = read_image(str(image_path), dump_output=SAVE_OUTPUTS_FOR_DEBUG)
152
153        # Preprocess #
154        input_data = preprocess_image(image, dump_output_image_name=image_name, dump_output=SAVE_OUTPUTS_FOR_DEBUG)
155
156        # Run inference #
157        output = inference_session.run(None, {input_name: input_data})
158
159        # Postprocess #
160        class_name, confidence = postprocess_output(output, labels, dump_output_image_name=image_name, dump_output=SAVE_OUTPUTS_FOR_DEBUG)
161
162        # Display results #
163        print(f"Class: {class_name} Confidence: {confidence * 100.0:.2f}%")
164
165
166if __name__ == "__main__":
167    parser = argparse.ArgumentParser(description="Classify an image using an ONNX ResNet50 model.")
168
169    parser.add_argument("-i", "--images_path", type=str, required=False, default=INPUT_IMAGES_PATH, help="Path to the image file.")
170    parser.add_argument("-m", "--model_path", type=str, required=False,  default=MODEL_PATH, help="Path to the image file.")
171    args = parser.parse_args()
172
173    # Setup output directory
174    if SAVE_OUTPUTS_FOR_DEBUG:
175        recreate_directory(OUTPUT_DEBUG_PATH)
176
177    main(images_path=args.images_path, model_path=args.model_path)

Note

Notice how the 5 stages of our application pipeline are described in lines 150-163

# Load image #
image, image_name = read_image(str(image_path), dump_output=SAVE_OUTPUTS_FOR_DEBUG)

# Preprocess #
input_data = preprocess_image(image, dump_output_image_name=image_name, dump_output=SAVE_OUTPUTS_FOR_DEBUG)

# Run inference #
output = inference_session.run(None, {input_name: input_data})

# Postprocess #
class_name, confidence = postprocess_output(output, labels, dump_output_image_name=image_name, dump_output=SAVE_OUTPUTS_FOR_DEBUG)

# Display results #
print(f"Class: {class_name} Confidence: {confidence * 100.0:.2f}%")

Note

Notice how key functions have the parameter dump_output: bool. This dumps binary files with the results of each function. These outputs will become our reference outputs as we build our GStreamer pipeline to validate if we have the correct expected values out of each GStreamer plugin.

To run the script:

sima-user@docker-image-id$ python src/x86_reference_app/resnet50_reference_classification_app.py
    Recreated directory: /home/docker/sima-cli/building_applications_mlsoc_1.4.0/src/x86_reference_app/debug

    Processing all images in: /home/docker/sima-cli/building_applications_mlsoc_1.4.0/src/x86_reference_app/../../data

    Processing image [0] --> Class: 207: 'golden retriever', Confidence: 98.38%

The resulting directory structure should look like:

sima-user@docker-image-id$ tree -L 4
    .
    ├── data
    │   ├── golden_retriever_207.jpg
    │   ├── imagenet_labels.txt
    │   └── openimages_v7_images_and_labels.pkl
    ├── models
    │   ├── download_resnet50.py
    │   └── resnet50_model.onnx
    ├── README.md
    ├── requirements.txt
    └── src
        ├── modelsdk_quantize_model
        │   └── resnet50_quant.py
        └── x86_reference_app
            ├── debug
            │   ├── golden_retriever_207_inference_output_probabilities.bin
            │   ├── golden_retriever_207_preprocessed_rgb_nhwc_fp32.bin
            │   └── golden_retriever_207_rgb.bin
            └── resnet50_reference_classification_app.py

Note

Notice how a newly created debug directory appears that has intermediate dumps of data we will use later in this guide.

Run on the MLSoC CPU (Arm A65)

To run the same resnet50_reference_classification_app.py application on the A65 of the MLSoC, simply:

  1. Replicate the same directory structure in a directory on the MLSoC devkit

    • scp the application, data and models directories to the board

  2. ssh into the board

  3. pip3 install onnxruntime if you have not already done so

  4. Run the application

Conclusion and next steps

In this section, we:

  • Developed a reference application that will serve as a guideline of what we need to develop in the MLSoC.

  • The reference application could be executed on a host machine or on the MSLoC.

  • The application also dumps each of the intermediate outputs which will be used during development to ensure we have the correct functional output out of our GStreamer plugins.

Next we will take our first step into MLSoC development: taking our trained model, and using the ModelSDK to compile and optimize it for running on the MLA.