Filesrc Tutorial

This tutorial shows how to use the Filesrc Pipeline for multi-model processing of local image files. You will learn how to run advanced computer vision tasks such as segmentation, detection, and anomaly analysis using YOLO, Teacher-Student distillation, and AutoEncoder networks on the SiMa.ai MLSoC platform.

Note

This section covers building filesrc pipelines with PePPi (Python).

Features:

  • Multi-model pipeline combining segmentation, detection, and anomaly analysis

  • Local file processing using filesrc input

  • Real-time anomaly detection using Teacher-Student and AutoEncoder networks

  • UDP streaming output for visualization

Purpose

This application demonstrates how to use the SiMa PePPi API to create a multi-model pipeline that:

  • Loads video frames from a local directory using filesrc

  • Uses a YOLO model to perform segmentation and detect objects of interest

  • Processes the detected regions using Teacher, Student, and AutoEncoder models

  • Calculates anomaly maps via mean squared error between Teacher-Student and AutoEncoder outputs

  • Combines these maps to generate predictions

  • Writes annotated results via UDP stream for visualization

Configuration Overview

All runtime settings are managed through project.yaml. Below are the parameters grouped by function.

Input/Output Configuration

Parameter

Description

Example

source.name

Input type; here using local image files

"filesrc"

source.value

Folder path containing input samples

"<Folder containing input samples>"

udp_host

Host IP to stream annotated output

"<HOST_IP>"

port

UDP port for the output stream

"<PORT_NUM>"

pipeline

Inference pipeline used

"AutoEncoderPipeline"

Model Configuration

YOLO Model (Index 0)

Parameter

Description

Value

name

Model name

"YOLO"

targz

Path to model archive

"yolo_seg_cls.tar.gz"

normalize

Whether to normalize input

true

channel_mean

Input mean values

[0.0, 0.0, 0.0]

channel_stddev

Input stddev values

[1.0, 1.0, 1.0]

padding_type

Padding method

"CENTER"

aspect_ratio

Maintain aspect ratio

false

topk

Maximum number of detections

10

detection_threshold

Detection score threshold

0.7

label_file

Label map file path

"labels.txt"

input_img_type

Image color format

"BGR"

Teacher / Student / AutoEncoder Models (Index 1–3)

Parameter

Description

Value (same across all)

name

Model name

"Teacher", "Student", "AutoEncoder"

targz

Model archive

teacher_int8_mpk.tar.gz, etc.

normalize

Enable input normalization

true

channel_mean

Input mean

[0.407, 0.446, 0.469]

channel_stddev

Input stddev

[0.289, 0.273, 0.277]

input_img_type

Input format

"BGR"

Project Configuration

project.yaml

 1source:
 2  name: "filesrc"
 3  value: "<Folder containing input samples>"
 4udp_host: "<HOST_IP>"
 5port: "<PORT_NUM>"
 6pipeline: "AutoEncoderPipeline"
 7
 8Models:
 9  - name: "YOLO"
10    targz: "yolo_seg_cls.tar.gz"
11    normalize: true
12    channel_mean: [0.0, 0.0, 0.0]
13    channel_stddev: [1.0, 1.0, 1.0]
14    padding_type: "CENTER"
15    aspect_ratio: false
16    topk: 10
17    detection_threshold: 0.7
18    label_file: "labels.txt"
19    input_img_type: "BGR"
20
21  - name: "Teacher"
22    targz: "teacher_int8_mpk.tar.gz"
23    normalize: true
24    channel_mean: [0.407, 0.446, 0.469]
25    channel_stddev: [0.289, 0.273, 0.277]
26    input_img_type: "BGR"
27
28  - name: "Student"
29    targz: "student_int8_mpk.tar.gz"
30    normalize: true
31    channel_mean: [0.407, 0.446, 0.469]
32    channel_stddev: [0.289, 0.273, 0.277]
33    input_img_type: "BGR"
34
35  - name: "AutoEncoder"
36    targz: "autoencoder_int8_mpk.tar.gz"
37    normalize: true
38    channel_mean: [0.407, 0.446, 0.469]
39    channel_stddev: [0.289, 0.273, 0.277]
40    input_img_type: "BGR"

Script Behavior

The main script performs the following:

  1. Initializes four SiMa MLSoC sessions for the YOLO, Teacher, Student, and AutoEncoder models

  2. Loads and loops through input frames using filesrc

  3. Runs the YOLO segmentation model to isolate regions of interest

  4. Passes the segmented region through the Teacher, Student, and AutoEncoder networks

  5. Computes two anomaly maps (Teacher vs. Student, AutoEncoder vs. Student)

  6. Normalizes and combines these maps to highlight regions of potential anomaly

  7. Post-processes the combined map and generates a final prediction

  8. Converts and streams the result via UDP using VideoWriter

Note

This pipeline requires multiple model files and is designed for advanced anomaly detection scenarios.

Model Details

Download Model

Note

The precompiled models are available for the Modalix devkit only for this pipeline.

Pipeline Architecture

Stage

Description

Input Processing

Loads frames from local directory using filesrc

YOLO Segmentation

Performs object detection and segmentation to identify regions of interest

Teacher Network

Processes segmented regions using pre-trained teacher model

Student Network

Processes same regions using student model for knowledge distillation

AutoEncoder Network

Reconstructs input regions to detect anomalies

Anomaly Detection

Computes mean squared error between Teacher-Student and AutoEncoder outputs

Output Generation

Combines anomaly maps and streams results via UDP

Build and Deploy

Follow the standard PePPi build and deploy process:

  1. Note:

    This multimodel/filesrc application demo works for Modalix target only.

  2. Prerequisites:

    1. Make sure the device is connected via SDK for deployment of project to device. Please follow the setup device section and follow the same.

sima-user@docker-image-id:/home/docker/sima-cli/workspace$ mpk device connect -t root@192.168.135.104
ℹ Connecting to root@192.168.135.104...
🔗 Connection established to 192.168.135.104 .
ℹ Fetching Device Plugin Version data file from : 192.168.135.104 ...
✔ Successfully fetched and updated Plugin Version Data file from : 192.168.135.104.

  3. Prepare the project directory:

    sima-user@docker-image-id:/home/docker/sima-cli/workspace$ mkdir filesrc_pipeline
sima-user@docker-image-id:/home/docker/sima-cli/workspace$ cd filesrc_pipeline/

  4. Copy the PCIe pipeline from SDK to workspace project directory :

    sima-user@docker-image-id:/home/docker/sima-cli/workspace/filesrc_pipeline$ cp -r /usr/local/simaai/app_zoo/Peppi/Multimodel-Demo/ .
sima-user@docker-image-id:/home/docker/sima-cli/workspace/filesrc_pipeline$ cd Multimodel-Demo/
sima-user@docker-image-id:/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo$ ls
constants.py  helper.py  main.py  mean_std.json  project.yaml  README.md

  5. Download required model files:

    # Download YOLO model
wget https://docs.sima.ai/pkg_download/appzoo/1.6/peppi/yolo_seg_cls.tar.gz

# Download Teacher model
wget https://docs.sima.ai/pkg_download/appzoo/1.6/peppi/teacher_int8_mpk.tar.gz

# Download Student model
wget https://docs.sima.ai/pkg_download/appzoo/1.6/peppi/student_int8_mpk.tar.gz

# Download AutoEncoder model
wget https://docs.sima.ai/pkg_download/appzoo/1.6/peppi/autoencoder_int8_mpk.tar.gz

sima-user@docker-image-id:/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo$ ls
constants.py  helper.py  main.py  mean_std.json  project.yaml  README.md  STAnomalyDet.tar.gz
sima-user@docker-image-id:/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo$ tar -xvf STAnomalyDet.tar.gz
./
./autoencoder_int8_mpk.tar.gz
./autoencoder_int8_stage2_a65.so
./student_int8_mpk.tar.gz
./teacher_int8_mpk.tar.gz
./yolo_seg_cls.tar.gz
./InputSamples/
./InputSamples/729.png
./InputSamples/input.png
./STAnomalyDet.tar.gz
sima-user@docker-image-id:/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo$ ls
autoencoder_int8_mpk.tar.gz       InputSamples   README.md                yolo_seg_cls.tar.gz
autoencoder_int8_stage2_a65.so    main.py        STAnomalyDet.tar.gz
constants.py                      mean_std.json  student_int8_mpk.tar.gz
helper.py                 project.yaml   teacher_int8_mpk.tar.gz

  6. Update Config parameters and Yaml :

    source:
  name: "filesrc"
  value: "InputSamples"
udp_host: "<Host_IP>"
port: "8005"
pipeline: "AutoEncoderPipeline"

Models:
  - name: "YOLO"
    targz: "yolo_seg_cls.tar.gz"
    channel_mean: [0.0, 0.0, 0.0]
    channel_stddev: [1.0, 1.0, 1.0]
    padding_type: "CENTER"
    aspect_ratio: false
    topk: 10
    detection_threshold: 0.7
    normalize: true
    label_file: labels.txt
    input_img_type: "BGR"

  - name: "Teacher"
    targz: "teacher_int8_mpk.tar.gz"
    normalize: true
    channel_mean: [0.407, 0.446, 0.469]
    channel_stddev: [0.289, 0.273, 0.277]
    input_img_type: "BGR"

  - name: "Student"
    targz: "student_int8_mpk.tar.gz"
    normalize: true
    channel_mean: [0.407, 0.446, 0.469]
    channel_stddev: [0.289, 0.273, 0.277]
    input_img_type: "BGR"

  - name: "AutoEncoder"
    targz: "autoencoder_int8_mpk.tar.gz"
    normalize: true
    channel_mean: [0.407, 0.446, 0.469]
    channel_stddev: [0.289, 0.273, 0.277]
    input_img_type: "BGR"

  7. Create the pipeline package:

    sima-user@docker-image-id:/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo$ mpk create --peppi -s . -d . --main-file=main.py --yaml-file=project.yaml
ℹ Generating requirements.txt file...
✔ Generated requirements.txt.
ℹ Dowloading required packages...
✔ Dowloaded required packages.
ℹ Building Rpm...
✔ Rpm built successfully.
ℹ Creating mpk file...
✔ Mpk file created successfully at
/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo/project.mpk

  8. Deploy the pipeline package:

    sima-user@docker-image-id:/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo$ mpk deploy -f project.mpk
ℹ Checking if App AutoEncoderPipeline Plugin Version Index File
/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo/AutoEncoderPipeline_plugin_version_index
.json exists...
❗ App AutoEncoderPipeline : File doesn't exist at
/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo/AutoEncoderPipeline_plugin_version_index
.json. Please check the path/file.
❔ Proceed by Skipping Plugin Version Check? [y/n]: y
‼ User chose to proceed with App AutoEncoderPipeline deployment, Skipping Plugin Version Check!
Performance could be sub-optimal!
🚀 Sending MPK to 192.168.135.104...
Transfer Progress for project.mpk:  100.00%
🏁 MPK sent successfully!
Installing MPK... ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  25%


After Deployment - STATUS -->

sima-user@docker-image-id:/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo$ mpk deploy -f project.mpk
ℹ Checking if App AutoEncoderPipeline Plugin Version Index File
/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo/AutoEncoderPipeline_plugin_version_index
.json exists...
❗ App AutoEncoderPipeline : File doesn't exist at
/home/docker/sima-cli/workspace/filesrc_pipeline/Multimodel-Demo/AutoEncoderPipeline_plugin_version_index
.json. Please check the path/file.
❔ Proceed by Skipping Plugin Version Check? [y/n]: y
‼ User chose to proceed with App AutoEncoderPipeline deployment, Skipping Plugin Version Check!
Performance could be sub-optimal!
🚀 Sending MPK to 192.168.135.104...
Transfer Progress for project.mpk:  100.00%
🏁 MPK sent successfully!
✔ MPK Deployed! ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
✔ MPK Deployment is successful for project.mpk.

  9. Visualize the results:

    On Host machine below commands need to be run -

GST_DEBUG=0 gst-launch-1.0 udpsrc port=8005 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! 'video/x-h264,stream-format=byte-stream,alignment=au' ! avdec_h264 ! fpsdisplaysink

** udpsrc port needs to be configured as per the project.yaml port configuration. **

Warning

Ensure all model files are downloaded and placed in the correct directory before building the pipeline package.