sima_utils.transformer.model.vision_language_model

Classes

VisionLanguageModel

Vision-language model implementation.

Module Contents

class sima_utils.transformer.model.vision_language_model.VisionLanguageModel

Vision-language model implementation.

static from_hf_cache(model_name: str, hf_cache_path: pathlib.Path | str, onnx_path: pathlib.Path | str, sima_path: pathlib.Path | str, max_num_tokens: int, system_prompt: str | None = None, override_language_group_size: int | None = None, override_language_group_offsets: list[int] | None = None, override_language_future_token_mask_size: int = 1) → VisionLanguageModel

Creates a VisionLanguageModel object from cached Hugging Face model.

Parameters:

  • model_name – Model name. This is used as a file name prefix for the generated onnx and model sdk files.

  • hf_cache_path – Path to the cached Hugging Face model.

  • onnx_path – Path to the generated ONNX files.

  • sima_path – Path to the generated SiMa files.

  • max_num_tokens – Maximum number of tokens, including both input and output tokens.

  • system_prompt – System prompt.

Returns:

A VisionLanguageModel object for file generation or evaluation.

gen_files(gen_mode: sima_utils.transformer.model.base.FileGenMode = FileGenMode.ALL, *, precision: sima_utils.transformer.model.base.FileGenPrecision | dict[str, sima_utils.transformer.model.base.FileGenPrecision] | None = None, log_level: int = logging.NOTSET, num_processes: int = 1, part: str | None = None, part_idx: int | None = None, resume: bool = False)

Generates files based on the provided file generation mode.

Parameters:

  • gen_mode – File generation mode.

  • precision – The precision to be used for Model SDK quantization mode.

  • log_level – Logging level.

  • part – Name of the part to be generated.

  • part_idx – Specific index of the part to be generated. For pre/post model, the index is the layer index; for cache model, the index is the token index.

  • resume – Generate the files only if it does not exist.

evaluate(eval_mode: sima_utils.transformer.model.base.EvalMode, query: str, image: pathlib.Path | str | numpy.ndarray | None) → str

Evaluates the model with the input query and the image in the specified mode.

Parameters:

  • eval_mode – Evaluation mode.

  • query – User query.

  • image – Path to the image or preprocessed image in numpy array.

Returns:

Generated output text.

run_model(eval_mode: sima_utils.transformer.model.base.EvalMode, ifms: list[numpy.ndarray]) → list[numpy.ndarray]

Runs the model based on the evaluation mode.

get_language_embeddings_tensor() → numpy.ndarray