simaaimask_filtering

This GStreamer plugin implements detessellation and segmentation mask composition for multi-object detection pipelines. It parses bounding boxes and mask tensors, detessellates tiled mask outputs, and combines them with the associated bounding boxes to generate composite results per object.

In other words, this plugin applies a mask to incoming video frames or inference results to filter or suppress undesired regions in the dataβ€”commonly for object detection, segmentation, or privacy filtering tasks.

Features

  • Detessellation of tiled feature maps for segmentation masks.

  • Per-object mask extraction using class-wise slicing.

  • Bounding box scaling and alignment to the original frame resolution.

  • Dequantization of mask logits with support for configurable scale/zero-point.

  • JSON-driven configuration for pipeline integration.

Components

UserContext Class

  • Initializes from a configuration JSON file.

  • Implements run() method invoked per frame.

  • Manages all input buffer decoding and output composition.

Utilities

  • calculate_slice_grid – Computes slicing indices for tensor slicing.

  • calculate_slice_grid_sizes – Calculates tile memory sizes based on dimensions and alignment.

  • calculate_slice_grid_offsets – Computes memory offsets for each tile.

  • reorder_data_from_16c_blocks – Reorders blocked channel-major data.

  • detessellate – Reconstructs full feature maps from tiled and sliced inputs.

Input Buffers

  • Masks Buffer: Quantized tiled mask logits.

  • Bounding Boxes Buffer: Stage-2 bounding boxes with confidence scores.

  • Both buffers must match configuration keys under buffers.input.

Output Buffer

  • Serialized vector of OutBox_Mask structures (bounding boxes + masks).

  • First 4 bytes indicate the number of valid entries.

Installation

  • Yocto SDK must be installed.

For manual installation:

scp libgstsimaaimask_filtering.so sima@<IP address of DaVinci board>:/data/my_pipeline/libs

Usage

To build the plugin use:

source /opt/poky/4.0.10/environment-setup-cortexa65-poky-linux
mkdir build && cd build && cmake .. && make

Get the correct mask from the MLA which returns a mask prediction to the proper class ID and concatenate it with the corresponding bounding box.

Example of usage:

GST_DEBUG=2 gst-launch-1.0 --gst-plugin-path=/data/simaai/applications/Mask_RCNN/lib simaaisrc location=/data/ev_kernel.out delay=1000 ! mask_filtering config="/data/mask_filtering_config.json" ! fakesink

Configuration

Configuration example:

{
  "version": 0.1,
  "node_name": "mask_filtering",
  "memory": {
    "cpu": 0, <================================================== NOT USED
    "next_cpu": 0 <============================================== Memory type used by allocator. 0 means A65 memory will be used
  },
  "system": {
    "out_buf_queue": 1,
    "debug": 0, <================================================ NOT USED
    "dump_data": 0 <============================================= Dump the output buffer into file at /tmp
  },
  "buffers": {
    "input": [
      {
        "name": "bbox", <======================================== Input buffer 0 name
        "size": 604     <======================================== Input buffer 0 size
      },
      {
        "name": "masks", <======================================= Input buffer 1 name
        "size": 28537600 <======================================= Input buffer 1 size
      }
    ],
    "output": {
      "size": 362400 <=========================================== Output buffer size
    }
  },
  "num_of_classes": 90 <========================================= Number of classes based on coco_labels.txt
}

Contributing

License

Acknowledgments