Uh_rdZddlmZddlmZmZmZmZddlZddl Zddlm Z ddl m Z m Z mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZddl m!Z!ddl"m#Z#ejHe%Z&eGddeZ'eGddeZ(Gdde jRZ*eGddeZ+edGdde+Z,GddeeZ-ed Gd!d"e+eZ.gd#Z/y)$zPyTorch PaliGemmamodel.) dataclass)ListOptionalTupleUnionN)nn)Cache HybridCache StaticCache)GenerationMixin)FlashAttentionKwargs)BaseModelOutputWithPast)PreTrainedModel)Unpack) LossKwargs ModelOutputauto_docstringcan_return_tupleis_torchdynamo_compilinglogging) AutoModel)PaliGemmaConfigc:eZdZUdZdZeejed<y)PaligemmaModelOutputWithPasta Base class for Paligemma outputs, with hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/paligemma/modeling_paligemma.pyrr&s88<%"3"34;r'rceZdZUdZdZeejed<dZ eejed<dZ ee e eje fed<dZeeejed<dZeeejed<dZeejed<y) PaliGemmaCausalLMOutputWithPasta Base class for PaliGemma causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rr r!r"r+rr#r$r%r,r-rrr r.rr/rr&r'r(r*r*Gs<)-D(5$$ %,*.FHU&& '.GKOXeD):):$;U$BCDK8image_featuresr.s r(forwardz$PaliGemmaMultiModalProjector.forwardts N3 r')rr r!rr7rC __classcell__r?s@r(r1r1osrrr'r1c@eZdZeZdZdZdgZdZdZ dZ dZ dZ dZ dZdZy)PaliGemmaPreTrainedModelTr1r-c\t|jd|jjj}t |t j rY|jjjd||j%|jjjyyy)Ninitializer_range)meanstd) getattrr2get_text_configrJ isinstancerr8weightdatanormal_r5zero_)r>modulerMs r( _init_weightsz&PaliGemmaPreTrainedModel._init_weightssdkk#6 8S8S8U8g8gh fbii ( MM   & &CS & 9{{&   &&(' )r'N)rr r!r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_flash_attn_2_supports_sdpa_supports_attention_backendrVr&r'r(rGrGzsJ"L&*#78"3  $!!N"&)r'rGz{ The Base Paligemma model which consists of a vision backbone and a language model withou language modeling head., ) custom_introc#eZdZddiZdeffd ZdZdZ ddee fdZ d e jfd Z ee dd e j d e jd ee j"d ee j deeee jefdee j dee j dee jdee j dee dee dee dee deedeeeffdZxZS)PaliGemmaModelzlanguage_model.modellanguage_modelr2ct||tj|j|_t ||_|jj|_ tj|j}||_ |jj|jjnd|_ |jy)N)r2)r6r7r from_configr9 vision_towerr1multi_modal_projector text_config vocab_sizerer2 pad_token_id post_init)r>r2rer?s r(r7zPaliGemmaModel.__init__s  %119M9MN%A&%I" ,,77"..f6H6HI,8< 8P8P8\DKK44bd r'c6|jjSrA)reget_input_embeddingsr>s r(rpz#PaliGemmaModel.get_input_embeddingss""7799r'c:|jj|yrA)reset_input_embeddingsr>values r(rsz#PaliGemmaModel.set_input_embeddingss 007r' is_trainingc`|jjjdk(r |d|vr|Sy||n |j}t |t }t j|jj}||}|jdd\} } |r|j} nUt |tr|j} n4t |t jr|jdn |d| zdz} ||jdk(r|St j| | f||j|j } | dk7r%|rt j"| d } n d| ddd| f<| t j$| |j |j'ddkDz} | ddddddfj)| ddd} || j+} |jd} |rd| t-d | ddddddd| fj/|ddddddfj1| j dk(d| ddddddd| f<| ddddddd| f|ddddddfj1| j z}|dk(}| ddddddd| fj/||| ddddddd| f<| S) Nflash_attention_2rKrrgrr fill_valuedtypedevicediagonalr}z/Token type ids must be provided during training)r2rk_attn_implementationtrainingrPr r#finfor|minshapeget_max_cache_shaper Tensordimfullr}triuarangereshapeexpandclone ValueError masked_fillto)r>attention_masktoken_type_idsr-cache_position input_tensorrvusing_static_cache min_dtypeinputs_lead_dimsequence_length target_length causal_mask mask_length padding_masks r(_update_causal_maskz"PaliGemmaModel._update_causal_masks ;; " " 7 7;N N)c^.C%%%0%!*C! !jj m ,$**]k]r]r  a #jjqA 36 A///0u||M.:O:OPSaSiSijlnoSppp !$a"23::?ArSUV  %%++-K(..r2K!)$%VWW5@Aq,;,AV5W5c5c"1dD!#3477 8J8JKqPRS6 Aq!\k\12 'q!Q  '<=qRVX\^_O_@`@c@cdodvdv@wwL'1,L1