sima_utils.transformer.prompt_template
======================================

.. py:module:: sima_utils.transformer.prompt_template


Attributes
----------

.. autoapisummary::

   sima_utils.transformer.prompt_template.DEFAULT_IMAGE_PLACEHOLDER_TOKEN_ID
   sima_utils.transformer.prompt_template.VLM_PROMPT_TEMPLATE
   sima_utils.transformer.prompt_template.arch


Classes
-------

.. autoapisummary::

   sima_utils.transformer.prompt_template.PromptFormatter
   sima_utils.transformer.prompt_template.Llama2PromptFormatter
   sima_utils.transformer.prompt_template.Llama3PromptFormatter
   sima_utils.transformer.prompt_template.LlavaPromptFormatter
   sima_utils.transformer.prompt_template.PaliGemmaPromptFormatter
   sima_utils.transformer.prompt_template.GemmaPromptFormatter
   sima_utils.transformer.prompt_template.Phi3PromptFormatter


Functions
---------

.. autoapisummary::

   sima_utils.transformer.prompt_template.multimodal_concat


Module Contents
---------------

.. py:data:: DEFAULT_IMAGE_PLACEHOLDER_TOKEN_ID
   :value: -200


.. py:data:: VLM_PROMPT_TEMPLATE

.. py:function:: multimodal_concat(text_token_ids: numpy.ndarray, vision_proj: numpy.ndarray, embed_weight: numpy.ndarray) -> numpy.ndarray

   Combine text and vision embedding tensors.

   :param text_token_ids: The text token ids with image placeholder.
   :type text_token_ids: np.ndarray
   :param vision_proj: The vision projection tensor.
   :type vision_proj: np.ndarray
   :param embed_weight: The embedding weight matrix.
   :type embed_weight: np.ndarray

   :returns: A tensor with combined text and vision embeddings.


.. py:class:: PromptFormatter(vlm_arch: str, system_message: str | None = None)

   Prompt formatter for VLM.



   .. py:attribute:: vlm_arch
      :type:  str


   .. py:attribute:: system_message
      :type:  str | None


   .. py:attribute:: image_placeholder_id
      :type:  int


   .. py:method:: set_system_message(msg: str)


   .. py:method:: formatted_prompt(query: str, has_image: bool = False) -> list[str]

      Format a query according to the prompt template.

      :param query: A text part of a user query.
      :param has_image: Whether the prompt includes a image placeholder token.

      :returns: The formatted query as a list of strings.



   .. py:method:: tokenize_prompt(tokenizer: sima_utils.transformer.llm_tokenizer.LlmTokenizer, messages: list) -> numpy.ndarray

      Tokenize a formatted prompt.

      :param tokenizer: The object of tokenizer needed.
      :type tokenizer: LlmTokenizer
      :param messages: The list of formatted query messages.
      :type messages: list

      :returns: An array of token ids with possible image placeholders.
                The first dimension indicates batch, the number of queries.



.. py:class:: Llama2PromptFormatter(vlm_arch: str, system_message: str | None = None)



   LLAMA 2 prompt formatter.



   .. py:attribute:: B_INST
      :type:  str
      :value: '[INST]'



   .. py:attribute:: E_INST
      :type:  str
      :value: '[/INST]'



   .. py:attribute:: B_SYS
      :type:  str
      :value: Multiline-String

      .. raw:: html

         <details><summary>Show Value</summary>

      .. code-block:: python

         """<<SYS>>
         """

      .. raw:: html

         </details>




   .. py:attribute:: E_SYS
      :type:  str
      :value: Multiline-String

      .. raw:: html

         <details><summary>Show Value</summary>

      .. code-block:: python

         """
         <</SYS>>
         
         """

      .. raw:: html

         </details>




   .. py:method:: formatted_prompt(query: str, has_image: bool = False) -> list[str]

      Format a query according to the prompt template.

      :param query: A text part of a user query.
      :param has_image: Whether the prompt includes a image placeholder token.

      :returns: The formatted query as a list of strings.



   .. py:method:: tokenize_prompt(tokenizer: sima_utils.transformer.llm_tokenizer.LlmTokenizer, messages: list[list[str]]) -> numpy.ndarray

      Tokenize a formatted prompt for LLAMA2.




.. py:class:: Llama3PromptFormatter(vlm_arch: str, system_message: str | None = None)



   LLAMA 3 prompt formatter.



   .. py:method:: tokenize_prompt(tokenizer: sima_utils.transformer.llm_tokenizer.LlmTokenizer, messages: list[list[str]]) -> numpy.ndarray

      Tokenize a formatted prompt for LLAMA3.




.. py:class:: LlavaPromptFormatter(vlm_arch: str, system_message: str | None = None)



   LLAVA prompt formatter.



   .. py:method:: tokenize_prompt(tokenizer: sima_utils.transformer.llm_tokenizer.LlmTokenizer, messages: list[list[str]]) -> numpy.ndarray

      Tokenize a formatted prompt for LLAVA.




.. py:class:: PaliGemmaPromptFormatter(vlm_arch: str, system_message: str | None = None)



   PaliGemma prompt formatter.



   .. py:method:: tokenize_prompt(tokenizer: sima_utils.transformer.llm_tokenizer.LlmTokenizer, messages: list[list[str]]) -> numpy.ndarray

      Tokenize a formatted prompt for PaliGemma.

              Template: <image> + <bos> + <prompt> + <
      >




.. py:class:: GemmaPromptFormatter(vlm_arch: str, system_message: str | None = None)



   GEMMA prompt formatter.



   .. py:attribute:: boi_id
      :type:  int


   .. py:attribute:: eoi_id
      :type:  int


   .. py:method:: tokenize_prompt(tokenizer: sima_utils.transformer.llm_tokenizer.LlmTokenizer, messages: list[list[str]]) -> numpy.ndarray

      Tokenize a formatted prompt for GEMMA.




.. py:class:: Phi3PromptFormatter(vlm_arch: str, system_message: str | None = None)



   PHI 3 prompt formatter.

   Chat format (https://huggingface.co/microsoft/Phi-3.5-mini-instruct)
   <|system|>
   You are a helpful assistant.<|end|>
   <|user|>
   How to explain Internet for a medieval knight?<|end|>
   <|assistant|>


   .. py:attribute:: B_SYS
      :type:  str
      :value: '<|system|>'



   .. py:attribute:: B_USER
      :type:  str
      :value: '<|user|>'



   .. py:attribute:: B_ASSISTANT
      :type:  str
      :value: '<|assistant|>'



   .. py:attribute:: END
      :type:  str
      :value: '<|end|>'



   .. py:method:: formatted_prompt(query: str, has_image: bool = False) -> list[str]

      Format a query according to the prompt template.

      :param query: A text part of a user query.
      :param has_image: Whether the prompt includes a image placeholder token.

      :returns: The formatted query as a list of strings.



   .. py:method:: tokenize_prompt(tokenizer: sima_utils.transformer.llm_tokenizer.LlmTokenizer, messages: list[list[str]]) -> numpy.ndarray

      Tokenize a formatted prompt for PHI3.




.. py:data:: arch