.. _developing_gstreamer_app_reference_app: Reference Classification Application #################################### In order to build an application on the MLSoC, we will first write a python application that will serve as a reference, and then begin to build our GStreamer application. In this first section, we have the code and resources for the python based application. Resources and setup =================== Please download and place this archive in your **Palette docker container**. .. button-link:: https://docs.sima.ai/pkg_downloads/SDK1.4.0/building_applications_mlsoc_1.4.0.tar.xz :color: primary :align: left :shadow: Download Reference App In your environment, uncompress the application, go into the uncompressed folder, make sure to install the requirements: .. code-block:: console sima-user@docker-image-id$ tar xvf building_applications_mlsoc_1.4.0.tar.xz building_applications_mlsoc_1.4.0/ building_applications_mlsoc_1.4.0/requirements.txt building_applications_mlsoc_1.4.0/data/ building_applications_mlsoc_1.4.0/data/imagenet_labels.txt building_applications_mlsoc_1.4.0/data/openimages_v7_images_and_labels.pkl building_applications_mlsoc_1.4.0/data/golden_retriever_207.jpg building_applications_mlsoc_1.4.0/README.md building_applications_mlsoc_1.4.0/models/ building_applications_mlsoc_1.4.0/models/download_resnet50.py building_applications_mlsoc_1.4.0/src/ building_applications_mlsoc_1.4.0/src/x86_reference_app/ building_applications_mlsoc_1.4.0/src/x86_reference_app/resnet50_reference_classification_app.py building_applications_mlsoc_1.4.0/src/modelsdk_quantize_model/ building_applications_mlsoc_1.4.0/src/modelsdk_quantize_model/resnet50_quant.py sima-user@docker-image-id$ cd building_applications_mlsoc_1.4.0/ sima-user@docker-image-id$ tree -L 3 . ├── data │ ├── golden_retriever_207.jpg │ ├── imagenet_labels.txt │ └── openimages_v7_images_and_labels.pkl ├── models │ └── download_resnet50.py ├── README.md ├── requirements.txt └── src ├── modelsdk_quantize_model │ └── resnet50_quant.py └── x86_reference_app └── resnet50_reference_classification_app.py sima-user@docker-image-id$ pip3 install -r requirements.txt .. note:: * ``data``: Any data we will use in the reference application or the quantization script * ``golden_retriever_207.jpg``: A test image we will use in our scripts. * ``imagenet_labels.txt``: The list of labels from ImageNet we will use to perform inference and retrieve what class was predicted. * ``openimages_v7_images_and_labels.pkl``: The labels and images from Open Images that we will use to calibrate our model during quantization. * ``models``: Where we will store our model * ``download_resnet50.py``: Downloads a pre-trained ResNet50 model, add a ``softmax`` node at the end, and saves the model. * The output of this script is the core model we will be using throughout the guide. * ``src``: Application and quantization scripts * ``modelsdk_quantize_model/resnet50_quant.py``: * Script used to quantize our ``resnet50_model.onnx`` model using SiMa.ai's Palette ModelSDK APIs (output from ``download_resnet50.py``). * ``x86_reference_app/resnet50_reference_classification_app.py``: * Our sample python application that can run on any machine and uses ``onnxruntime`` to run the ``resnet50_model.onnx`` along with all pre and post processing. * his will be our reference application, and we will be developing this exact application on the MLSoC. Python example for x86/Mac ========================== Download and save the model --------------------------- Throughout this guide, we will be using a ResNet50 model from TorchVision and we will add a ``softmax`` to the output. To download the model and add the ``softmax`` operator, the output will be stored in the directory ``models/resnet50_model.onnx``: .. code-block:: console sima-user@docker-image-id$ python models/download_resnet50.py Downloading: "https://download.pytorch.org/models/resnet50-11ad3fa6.pth" to /root/.cache/torch/hub/checkpoints/resnet50-11ad3fa6.pth 100%|███████████████████████████████████████████████████████████████████████████████████████████| 97.8M/97.8M [00:00<00:00, 114MB/s] ============== Diagnostic Run torch.onnx.export version 2.0.1+cpu ============== verbose: False, log level: Level.ERROR ======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ======================== Model exported successfully to /home/docker/sima-cli/building_applications_mlsoc_1.4.0/models/resnet50_export.onnx Simplified model saved to /home/docker/sima-cli/building_applications_mlsoc_1.4.0/models/resnet50_model.onnx console$ tree -L3 . ├── data │ ├── golden_retriever_207.jpg │ ├── imagenet_labels.txt │ └── openimages_v7_images_and_labels.pkl ├── models │ ├── download_resnet50.py │ └── resnet50_model.onnx ├── README.md ├── requirements.txt └── src ├── modelsdk_quantize_model │ └── resnet50_quant.py └── x86_reference_app └── resnet50_reference_classification_app.py Run the reference application ----------------------------- The following Python script demonstrates how to classify images using an ONNX ResNet50 model and ``onnxruntime``. This script can be run on any x86/Mac machine with the necessary dependencies installed. The reference application itself: .. code-block:: python :caption: resnet50_reference_classification_app.py :linenos: #************************************************************************** #|| SiMa.ai CONFIDENTIAL || #|| Unpublished Copyright (c) 2023-2024 SiMa.ai, All Rights Reserved. || #************************************************************************** # NOTICE: All information contained herein is, and remains the property of # SiMa.ai. The intellectual and technical concepts contained herein are # proprietary to SiMa and may be covered by U.S. and Foreign Patents, # patents in process, and are protected by trade secret or copyright law. # # Dissemination of this information or reproduction of this material is # strictly forbidden unless prior written permission is obtained from # SiMa.ai. Access to the source code contained herein is hereby forbidden # to anyone except current SiMa.ai employees, managers or contractors who # have executed Confidentiality and Non-disclosure agreements explicitly # covering such access. # # The copyright notice above does not evidence any actual or intended # publication or disclosure of this source code, which includes information # that is confidential and/or proprietary, and is a trade secret, of SiMa.ai. # # ANY REPRODUCTION, MODIFICATION, DISTRIBUTION, PUBLIC PERFORMANCE, OR PUBLIC # DISPLAY OF OR THROUGH USE OF THIS SOURCE CODE WITHOUT THE EXPRESS WRITTEN # CONSENT OF SiMa.ai IS STRICTLY PROHIBITED, AND IN VIOLATION OF APPLICABLE # LAWS AND INTERNATIONAL TREATIES. THE RECEIPT OR POSSESSION OF THIS SOURCE # CODE AND/OR RELATED INFORMATION DOES NOT CONVEY OR IMPLY ANY RIGHTS TO # REPRODUCE, DISCLOSE OR DISTRIBUTE ITS CONTENTS, OR TO MANUFACTURE, USE, OR # SELL ANYTHING THAT IT MAY DESCRIBE, IN WHOLE OR IN PART. # #************************************************************************** import cv2 import shutil import argparse import numpy as np import onnxruntime as ort from pathlib import Path from typing import Union # Constants ROOT_PATH = Path(__file__).parent.resolve() INPUT_IMAGES_PATH = str(Path(ROOT_PATH, "../../data/")) IMAGENET_LABELS_PATH = str(Path(ROOT_PATH, "../../data/imagenet_labels.txt")) MODEL_PATH = str(Path(ROOT_PATH, "../../models/resnet50_model.onnx")) SAVE_OUTPUTS_FOR_DEBUG = True OUTPUT_DEBUG_PATH = ROOT_PATH/"debug" #################### # Helper Functions # #################### def recreate_directory(dir_path: Union[Path, str]): # Ensure dir_path is a Path object if given a string dir_path = Path(dir_path) # Check if the directory exists if dir_path.exists(): # Remove the directory and all its contents shutil.rmtree(dir_path) print(f"Deleted directory: {dir_path}") # Recreate the directory dir_path.mkdir(parents=True, exist_ok=True) print(f"Recreated directory: {dir_path}") ################### # Setup Functions # ################### def setup(model_path: str, labels_path: str): # Load labels and ONNX model with open(labels_path, "r") as f: labels = [line.strip() for line in f.readlines()] # Load the ONNX model session = ort.InferenceSession(model_path) input_name = session.get_inputs()[0].name return labels, session, input_name ###################### # Pipeline Functions # ###################### def read_image(image_path: str, dump_output: bool = False): image = cv2.imread(str(image_path), cv2.IMREAD_UNCHANGED) rgb_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) image_name = Path(image_path).stem if dump_output: debug_image_name = image_name + "_rgb.bin" rgb_image.tofile(str(OUTPUT_DEBUG_PATH/debug_image_name)) return rgb_image, image_name # Function to preprocess the image def preprocess_image(image: np.ndarray, input_shape: tuple = (224, 224), scale_factor: tuple = 255.0, dump_output_image_name: str = None, dump_output: bool = False): mean = [0.485, 0.456, 0.406] stddv = [0.229, 0.224, 0.225] resized_image = cv2.resize(image, input_shape) image_data = resized_image / scale_factor image_data = (image_data - mean) / stddv if dump_output: # Dump reference output before transposing because MLA uses NHWC format not NCHW assert dump_output_image_name is not None debug_image_path = OUTPUT_DEBUG_PATH/f"{dump_output_image_name}_preprocessed_rgb_nhwc_fp32.bin" image_data.tofile(debug_image_path) image_data = np.transpose(image_data.astype('float32'), (2, 0, 1)) # Change to (C, H, W) image_data = np.expand_dims(image_data, axis=0) # Add batch dimension return image_data # Function to post-process the output def postprocess_output(output: np.ndarray, labels: dict, dump_output_image_name: str = None, dump_output: bool = False): probabilities = output[0][0] max_idx = np.argmax(probabilities) if dump_output: # Dump reference output before transposing because MLA uses NHWC format not NCHW assert dump_output_image_name is not None debug_image_path = OUTPUT_DEBUG_PATH/f"{dump_output_image_name}_inference_output_probabilities.bin" probabilities.tofile(debug_image_path) return labels[max_idx], probabilities[max_idx] def main(images_path: str, model_path: str, labels_path: str = IMAGENET_LABELS_PATH): """ Runs an application pipeline composed of the following 5 stages: Load image -> Preprocess image -> Run ResNet50 (inference) -> Postprocess outputs -> Display results """ # Initialization # labels, inference_session, input_name = setup(model_path=model_path, labels_path=labels_path) image_paths = list(Path(images_path).glob("*.jpg")) # 5 Stage application pipeline # print(f"\nProcessing all images in: {images_path}\n") for idx, image_path in enumerate(image_paths): print(f"Processing image [{idx}] --> ", end="") # Load image # image, image_name = read_image(str(image_path), dump_output=SAVE_OUTPUTS_FOR_DEBUG) # Preprocess # input_data = preprocess_image(image, dump_output_image_name=image_name, dump_output=SAVE_OUTPUTS_FOR_DEBUG) # Run inference # output = inference_session.run(None, {input_name: input_data}) # Postprocess # class_name, confidence = postprocess_output(output, labels, dump_output_image_name=image_name, dump_output=SAVE_OUTPUTS_FOR_DEBUG) # Display results # print(f"Class: {class_name} Confidence: {confidence * 100.0:.2f}%") if __name__ == "__main__": parser = argparse.ArgumentParser(description="Classify an image using an ONNX ResNet50 model.") parser.add_argument("-i", "--images_path", type=str, required=False, default=INPUT_IMAGES_PATH, help="Path to the image file.") parser.add_argument("-m", "--model_path", type=str, required=False, default=MODEL_PATH, help="Path to the image file.") args = parser.parse_args() # Setup output directory if SAVE_OUTPUTS_FOR_DEBUG: recreate_directory(OUTPUT_DEBUG_PATH) main(images_path=args.images_path, model_path=args.model_path) .. note:: Notice how the 5 stages of our application pipeline are described in :code:`lines 150-163` .. code-block:: python # Load image # image, image_name = read_image(str(image_path), dump_output=SAVE_OUTPUTS_FOR_DEBUG) # Preprocess # input_data = preprocess_image(image, dump_output_image_name=image_name, dump_output=SAVE_OUTPUTS_FOR_DEBUG) # Run inference # output = inference_session.run(None, {input_name: input_data}) # Postprocess # class_name, confidence = postprocess_output(output, labels, dump_output_image_name=image_name, dump_output=SAVE_OUTPUTS_FOR_DEBUG) # Display results # print(f"Class: {class_name} Confidence: {confidence * 100.0:.2f}%") .. note:: Notice how key functions have the parameter ``dump_output: bool``. This dumps binary files with the results of each function. These outputs will become our reference outputs as we build our GStreamer pipeline to validate if we have the correct expected values out of each GStreamer plugin. To run the script: .. code-block:: console sima-user@docker-image-id$ python src/x86_reference_app/resnet50_reference_classification_app.py Recreated directory: /home/docker/sima-cli/building_applications_mlsoc_1.4.0/src/x86_reference_app/debug Processing all images in: /home/docker/sima-cli/building_applications_mlsoc_1.4.0/src/x86_reference_app/../../data Processing image [0] --> Class: 207: 'golden retriever', Confidence: 98.38% The resulting directory structure should look like: .. code-block:: console sima-user@docker-image-id$ tree -L 4 . ├── data │ ├── golden_retriever_207.jpg │ ├── imagenet_labels.txt │ └── openimages_v7_images_and_labels.pkl ├── models │ ├── download_resnet50.py │ └── resnet50_model.onnx ├── README.md ├── requirements.txt └── src ├── modelsdk_quantize_model │ └── resnet50_quant.py └── x86_reference_app ├── debug │ ├── golden_retriever_207_inference_output_probabilities.bin │ ├── golden_retriever_207_preprocessed_rgb_nhwc_fp32.bin │ └── golden_retriever_207_rgb.bin └── resnet50_reference_classification_app.py .. note:: Notice how a newly created ``debug`` directory appears that has intermediate dumps of data we will use later in this guide. Run on the MLSoC CPU (Arm A65) ============================== To run the same ``resnet50_reference_classification_app.py`` application on the A65 of the MLSoC, simply: #. Replicate the same directory structure in a directory on the MLSoC devkit * `scp` the application, :code:`data` and :code:`models` directories to the board #. :code:`ssh` into the board #. :code:`pip3 install onnxruntime` if you have not already done so #. Run the application Conclusion and next steps ========================= In this section, we: * Developed a reference application that will serve as a guideline of what we need to develop in the MLSoC. * The reference application could be executed on a host machine or on the MSLoC. * The application also dumps each of the intermediate outputs which will be used during development to ensure we have the correct functional output out of our GStreamer plugins. Next we will take our first step into MLSoC development: taking our trained model, and using the :ref:`modelsdk` to compile and optimize it for running on the MLA.