quantize

Once the original trained model has been imported into the loaded_net format using the load_model() API, it can be quantized using the afe.apis.loaded_net.LoadedNet.quantize() API. The SiMa.ai software and silicon currently support 8-bit and 16-bit integer operations on the Machine Learning Accelerator (MLA) and floating-point operations on the Application Processing Unit (APU) and Computer Vision Unit (CVU).

In general, pre and post-processing functions will be implemented on the APU and CVU whilst model layers such as convolution, pooling, etc. will run on the MLA.

The partitioning across compute units will be done automatically by the quantizer and only the parts that run on the MLA will be quantized.

Mixed-precision Quantization

To maximize a compiled model’s execution speed while achieving a chosen accuracy, the ModelSDK supports quantizing an FP32 model into mixed INT8 and INT16 precision. To quantize a model with mixed precision you need to add layers/operators from your models that you are targeting for INT16 quantization. By default ModelSDK quantizes FP32 to INT8. Therefore, the remaining layers/operators will quantize to INT8.

To add layers to the INT16 quantization scheme you will need to call with_custom_quantization_configs from default_quantization and add INT16 targeted layers to quantization_config parameter of the quantize API.

In this form of quantization, the software automatically quantizes a model in different configurations and measures its accuracy, and then returns the best candidate found. This option is best used when the desired accuracy is between the accuracy achieved with INT8 and the accuracy achieved with INT16.

The function afe.apis.loaded_net.LoadedNet.quantize_with_accuracy_feedback() is intended to minimize the number of 16-bit nodes while reaching the target accuracy. Failing to reach the target accuracy indicates an error, however, minor accuracy losses may occur due to the statistical nature of the accuracy calculation.

model = loaded_net.quantize_with_accuracy_feedback(
calibration_data, evaluation_dataset, quant_configs,
accuracy_score=accuracy, target_accuracy=0.89)

Below is an example code for resnet50 model to add in your code before the quantize() API call.

from afe.core.configs import QuantizationPrecision

# Quantize the loaded net setting quantization precision of some nodes to int16.
# User can provide the custom quantization config dictionary in the following form:
#   custom_config_dict = {'node_name': {'quantization_config_field': value}}
# For example, in order to set the precision of a certain node to int16, the user needs
# to provide the config as follows:
#   custom_config_dict = {'node_name': {'quantization_precision': QuantizationPrecision.INT_16}}
# User must get the node names from the SiMaIR graph.  To obtain node names from
# SiMaIR graph, user needs to run int8 quantization and get the names of nodes
# examining the .sima.json in netron.  It is user's responsibility to decide which
# nodes should be quantized with higher precision.

int16_nodes = [
    "MLA_0/conv2d_add_relu_0", "MLA_0/max_pool2d_1", "MLA_0/conv2d_add_relu_2",
    "MLA_0/conv2d_add_relu_3", "MLA_0/conv2d_add_4", "MLA_0/conv2d_add_5"
]
quant_configs = default_quantization.with_custom_quantization_configs(
    {int16_node: {'quantization_precision': QuantizationPrecision.INT_16} for int16_node in int16_nodes}
)

sdk_net = loaded_net.quantize(
    calibration_data=calibration_data,
    quantization_config=quant_configs,
    model_name="tut_resnet50"
)

You can verify whether the above layers/operators are added to INT16 by executing and saving the sdk_net output. As shown below in tut_resnet_tensorflow.sima.json file search for any layer that you added in the INT16_nodes list. You can see that in output, the input_int16 will be True for that layer/operator.

sima-user@docker-image-id:model_zoo/resnet50/sdk_net_manual_mixed$ ls -ltr
    total 100536
    -rw-r--r-- 1 sima-user sima-user 388338 Aug  1 15:18 tut_resnet_tensorflow.sima.json
    -rw-r--r-- 1 sima-user sima-user 102552052 Aug  1 15:18 tut_resnet_tensorflow.sima
sima-user@docker-image-id:model_zoo/resnet50/sdk_net_manual_mixed$ vi tut_resnet_tensorflow.sima.json
    "class_name": "MLA_0/Conv2DAddActivation_5",
                            "name": "MLA_0/conv2d_add_5",
                            "inbound_nodes": [
                            ...
                            ...
                                "activ_attrs": null,
                                "input_int16": true,
                                "msb_left_shift": true,

TFLite Pre-Quantized Support

The ModelSDK is able to ingest a model that was previously quantized using TFLite. The model is first imported using the load_model() API function and then converted using .convert_to_sima_quantization() to prepare for compiling. Any floating-point computation in the model will remain floating-point and will not run on the MLA.

This function converts a loaded pre-quantized model into a Model, using the quantization that was provided in the input model. The model should have been quantized prior to loading. Any floating-point computation in the model will remain floating-point and will not run on the MLA.

LoadedNet.convert_to_sima_quantization(*, requantization_mode: RequantizationMode = RequantizationMode.sima, model_name: str | None = None, log_level: int | None = logging.NOTSET) → Model

Parameters:

requantization_mode – Determines what set of arithmetic approximations to use in ModelSDK
model_name – If given, is used as the name of the returned model.
log_level – Default is logging.NOTSET. Sets the logging level for this API call as described in the Logging section.

Module:

LoadedNet

Exceptions raised:

A pre-quantized operator cannot be converted to a SiMa quantized operator.

This example shows how to load and convert a pre-quantized TFlite model that has a single input named input_1 of shape 1,224,224,3:

# imports
from afe.load.importers.general_importer import ImporterParams, tflite_source
from afe.apis.loaded_net import load_model
from afe.ir.tensor_type import ScalarType
from afe.ir.defines import QuantizationFlavor

model_path='path/to/quantized/tflite/model'

# input shapes dictionary - NHWC format
# each key,value pair defines an input name and its shape
input_shapes = {'input_1': (1,224,224,3)}


# input types dictionary
# each key,value pair defines an input and its type
input_types = {'input_1': ScalarType.uint8}

# importer parameters
importer_params: ImporterParams = tflite_source(model_path=model_path,
                                                shape_dict=input_shapes,
                                                dtype_dict=input_types)

# load the model
loaded_net = load_model(importer_params)

# Convert quantized model - default is QuantizationFlavor.sima
quant_model = loaded_net.convert_to_sima_quantization()

# or use TFLite flavor..
#quant_model = loaded_net.convert_to_sima_quantization(quantization_flavor=QuantizationFlavor.tflite)



# save the quantized model
quant_model.save(model_name = model_name,
                output_directory = saved_model_directory)

The ModelSDK generally uses different quantized arithmetic than in other frameworks. Conversion will introduce differences due to rounding and other approximations. Two main quantization “flavors” are supported:

TFLite Quantization	Description
QuantizationFlavor.sima (default)	An efficient form of arithmetic that may sometimes sacrifice accuracy. Higher performance with possible loss of precision.
QuantizationFlavor.tflite	Performs arithmetic as described by TFlite documentation.It involves the operations multiply, shift, round, and add. Less performance but may be more accurate.

It is recommended to try using the default quantization flavor, and if the model’s accuracy is not sufficient, then use QuantizationFlavor.tflite.

Per-Layer Quantization Statistics

Given the FP32 and quantized versions of an ML network, ModelSDK has the ability to report per-layer quantization statistics. The API function to invoke this feature is shown below:

Model.analyze_quantization_error(evaluation_data: Iterable[InputValues], error_metric: Metric, *, local_feed: bool, log_level: int | None = logging.NOTSET)

Module:: Model

This function is useful for determining how each layer of the quantized model contributes to the overall error.

Parameters:

evaluation_data – A Python Iterable that provides data samples for the error analysis.
error_metric – String. Defines the error calculation method: * mae: Mean absolute error * mse: Mean squared error * psnr: Peak signal-to-noise ratio
local_feed – Boolean. See description below.
log_level – Default is logging.NOTSET. Sets the logging level for this API call as described in the Logging section.
simulated_arm – Do not use, leave at default value.

# load FP model
loaded_net = load_model(params)

# quantize
quant_model = loaded_net.quantize(calibration_data=calib_data,
                                quantization_config=default_quantization,
                                model_name=model_name)


# run per-layer analysis
quant_model.analyze_quantization_error(evaluation_data=test_data,
                                    error_metric='mae',
                                    local_feed=True)


# save
quant_model.save(model_name = model_name,
                output_directory = saved_model_directory)

Once the model is saved to disk, the _model_name_.sima.json file will contain the error values for each layer. These can be seen by examining the JSON file with a text editor and searching for the “layer Statistics” string:

"_status": "SIMA_QUANTIZED",
"Layer Statistics": {
    "metric": "mae",
    "error_value": 3.9165286500938237,
    "__decode_key__": "LayerStats"
},

..or by opening the JSON file with Netron, click on the layer in model digaram and view the ‘Layer Statistics’ field:

Note

It is not currently possible to run per-layer quantization statistics on a quantized model that has first been saved to disk and then loaded. It must be run on a model that has just been quantized using the .quantize() API.

local_feed=True: The floating-point model is passed the evaluation data as defined by the evaluation_data argument. The input of each node in the quantized network is fed the output data from the preceding node in the floating-point network and the output of the quantized node is dequantized and compared to the output of the equivalent node in the floating-point network - the error value is calculated as per the chosen metric. The displayed error value is the error value for that particular node only,
local_feed=False: The quantized model and the floating-point model that it was generated from are passed the same identical set of evaluation data as defined by the evaluation_data argument. The output of each node of the quantized model is dequantized and compared to the output of the corresponding node in the floating-point model - the error_value in the JSON file is the error of cumulative error of the quantized network up to that particular node (calculated as per the chosen metric). The error_value shown in the last node will be the cumulative error value of the entire quantized network.

Executing Loaded and Quantized Models

With the afe.apis.loaded_net.LoadedNet.execute() API, the ModelSDK allows quantized models to be executed in software with user-supplied input data so that the results can be evaluated. The quantized model is executed using a software implementation of the quantized operators. The model is executed with a single set of input tensor values and the output tensor values are returned. It does not simulate execution on the processor.

The ModelSDK allows users to evaluate the accuracy of quantized models using the afe.apis.model.Model.execute() API.

(Default method) Bit-accurate evaluation using Numpy routines in ML kernels - specify fast_mode=False in Model.execute() API.

(Second method) Bit-approximate evaluation using Tensorflow-optimized code - specify fast_mode=True in Model.execute() API.

The following code shows an example of how to use the afe.apis.model.Model.execute() API. In this example we have a numpy file of pre-processed data samples which are fed into the quantized model. The model has a single input input_1 and a single output:

Example Code

# quantize
quant_model = loaded_net.quantize(calibration_data=calib_data,
                                quantization_config=quant_configs,
                                model_name=model_name)


# read numpy file of preprocessed data & labels
dataset_f = np.load('preprocessed_data.npz')
test_data= dataset_f['x']
labels = dataset_f['y']


inputs = dict()
fp32_correct = 0
quant_correct = 0
# iterate over all test data
for i in range(test_images):

    # key=input name, value=data sample
    inputs['input_1'] = data[i]

    # execute quantized model - returns a list of ndarray
    tvm_fp32_model_out = loaded_net.execute(inputs)
    quant_model_out = quant_model.execute(inputs, fast_mode=True)

    # argmax
    fp_32prediction = np.argmax(tvm_fp32_model_out[0])
    quant_prediction = np.argmax(quant_model_out[0])

    # check output
    if fp_32prediction == labels[i]: fp32_correct+=1
    if quant_prediction == labels[i]: quant_correct+=1

(Third method) Bit-accurate evaluation of a quantized model in Accelerator Mode. This method executes a model on the SiMa development kit which must be connected to a network.

Model.execute_in_accelerator_mode(input_data: Iterable[InputValues], devkit: str, *, username: str = cp.DEFAULT_USERNAME, password: str = '', batch_size: int = 1, compress: bool = True, tessellate_parameters: TessellateParameters | None = None, log_level: int | None = logging.NOTSET, l2_caching_mode: L2CachingMode = L2CachingMode.NONE) → List[np.ndarray]

Module:: Model

Invoke the function Model.execute_on_accelerator_mode() with a list of pre-processed images. The API first compiles the model into an .lm file, and then obtains inference results by running the .lm file on the development kit.

Parameters:

input_data – The model is executed with a single set of input tensor values and the output tensor values are returned.
devkit – The URL for the development kit.
username – User login credential on the development kit.
password – User login credential on the development kit.
batch_size – Batch size to use in the compiled model.
compress – A flag to enable/disable DRAM data compression in the compiled LM file.
log_level – This sets the logging level for this call as described in the Logging section.
tessellate_parameters – Reserved parameter and takes default value.
l2_caching_mode – Reserved parameter and takes default value.

Example Code

    import os

    import cv2
    import numpy as np
    import argparse
    import time
    import pickle as pkl
    from typing import Any
    from afe.apis.defines import default_quantization
    from afe.apis.loaded_net import load_model
    from afe.core.utils import convert_data_generator_to_iterable, length_hinted
    from afe.ir.tensor_type import ScalarType
    from afe.load.importers.general_importer import tensorflow_source
    from afe.apis.release_v1 import DataGenerator
    from afe.apis.error_handling_variables import enable_verbose_error_messages

    """
    Script for quantizing, compiling and running model in accelerator mode.
    """

    MODEL_DIR = './UR_pb_resnet50-v1.5_fp32_224_224.pb'
    DATASET_DIR = './imagenet/val224.pkl'
    OUTPUT_DIR = './output'

    DAVINCI_HOSTNAME = ""
    DAVINCI_USERNAME = "sima"
    DAVINCI_PASSWORD = ""

    NUM_SAMPLES = 10


    def pre_process_func(image: np.ndarray) -> np.ndarray:
    from tensorflow.keras.applications.imagenet_utils import preprocess_input

    img = image.transpose([1, 2, 0]).astype(np.float32)
    img = preprocess_input(img)[:, :, ::-1]
    img = cv2.resize(img, (224, 224))
    return img


    def load_pickle(path: str) -> Any:
    with open(path, 'rb') as f:
    v = pkl.load(f)
    return v


    def _create_calibration_data():
    dataset = load_pickle(DATASET_DIR)
    input_generator = DataGenerator({"input_tensor": dataset['data']})
    # Apply the preprocessing function to the dataset
    input_generator.map({"input_tensor": pre_process_func})
    input_iterable = convert_data_generator_to_iterable(input_generator,
                                                    length_limit=NUM_SAMPLES)
    return input_iterable


    def main(model_path: str, hostname: str, username: str, password: str, enable_verbose_message: bool):
    print("SiMa ModelSDK tutorial example of running ResNet50 model in Accelerator Mode")

    if enable_verbose_message:
        enable_verbose_error_messages()

    os.makedirs(OUTPUT_DIR, exist_ok=True)
    model_name = os.path.split(model_path)[1]
    model_prefix = os.path.splitext(model_name)[0]

    # Models importer parameters
        input_name = "input_tensor"
        output_names = ["resnet_model/dense/BiasAdd"]
        input_shape = (1, 224, 224, 3)
        input_dtype = ScalarType.float32
        input_shapes_dict = {input_name: input_shape}

    # refer to the SDK User Guide for the specific format
        importer_params = tensorflow_source(model_path, input_shapes_dict, output_names)
        loaded_net = load_model(importer_params)

    # quantization with real training data samples
    # Note that quantization data is ALWAYS NHWC
        calibration_data = _create_calibration_data()

        print("Quantizing the model.")
        model_sdk_net = loaded_net.quantize(length_hinted(NUM_SAMPLES, calibration_data),
                default_quantization,
                model_name=model_prefix,
                arm_only=False)

    saved_model_directory = "sdk"
    model_sdk_net.save(model_name=model_name, output_directory=saved_model_directory)

        print("Executing quantized model in accelerator mode.")
        input_data = _create_calibration_data()
        start_time = time.time()
        outputs = model_sdk_net.execute_in_accelerator_mode(
        input_data=length_hinted(NUM_SAMPLES, input_data),
        devkit=hostname,
        username=username,
        password=password)
        end_time = time.time()
        print(f"Model is executed in accelerator mode. Elapsed time: {end_time - start_time} [s].")

        print("End of tutorial example of ResNet50 in Accelerator Mode.")


    if __name__ == "__main__":
        parser = argparse.ArgumentParser(description="Script with optional arguments")
        parser.add_argument('--model_path', type=str, default=MODEL_DIR)
        parser.add_argument('--hostname', type=str, default=DAVINCI_HOSTNAME)
        parser.add_argument('--username', type=str, default=DAVINCI_USERNAME)
        parser.add_argument('--password', type=str, default=DAVINCI_PASSWORD)
        parser.add_argument('--enable_verbose_message', action='store_true', help="Enable verbose messages")
        args = parser.parse_args()
        main(args.model_path, args.hostname, args.username, args.password, args.enable_verbose_message)