Uh ddlZddlmZddlmZddlmZddlmZm Z m Z m Z ddl Z ddl mZddlmZddlmZmZmZdd lmZdd lmZdd lmZmZdd lmZmZdd lm Z m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0m1Z1e(r ddl2m3Z3ddl4m5Z5e*jle7Z8eGddeZ9eGdde%Z:GddejvZ<GddejzZ>Gdd ejzZ?Gd!d"ejzZ@d#ZAdFd$ZBd%e jd&eDd'e jfd(ZE dGd)ejzd*e jd+e jd,e jd-e e jd.eFd/e eFd0e eFd'e e je jffd1ZGGd2d3ejzZHGd4d5ejzZIe&Gd6d7e!ZJe&Gd8d9eJZKe&Gd:d;eJeZLGd<d=ejzZMe&d>?Gd@dAeJZNe&dB?GdCdDeJeZOgdEZPy)HN)Callable) dataclass)partial)ListOptionalTupleUnion)ACT2FN)Cache HybridCache StaticCache)GenerationMixin)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tupleis_torch_flex_attn_availableis_torchdynamo_compilinglogging)deprecate_kwarg) AutoModel) Gemma3ConfigGemma3TextConfig) BlockMask)make_flex_block_causal_maskc:eZdZUdZdZeejed<y)Gemma3ModelOutputWithPasta Base class for Gemma3 outputs, with hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) __name__ __module__ __qualname____doc__r(rtorch FloatTensor__annotations__|/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/gemma3/modeling_gemma3.pyr'r'=s88<%"3"34;r1r'ceZdZUdZdZeejed<dZ eejed<dZ ee e eje fed<dZeeejed<dZeeejed<dZeejed<y) Gemma3CausalLMOutputWithPasta Base class for Gemma3 causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr()r)r*r+r,r5rr-r.r/r6r7r rr r8rr9r(r0r1r2r4r4^s<)-D(5$$ %,*.FHU&& '.GKOXeD):):$;U$BCDK8r? __class__s r2rDz&Gemma3TextScaledWordEmbedding.__init__s3  D ]ELL,ERWXr1 input_idsct|||jj|jj zSN)rCforwardr?toweightdtype)rGrIrHs r2rLz%Gemma3TextScaledWordEmbedding.forwards2wy)D,<,<,?,? @Q@Q,RRRr1)?) r)r*r+r,intfloatrDr-TensorrL __classcell__rHs@r2r;r;sGYsY3YSY_dYSSSr1r;c*eZdZdeffd ZdZxZS) Gemma3MLPconfigct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFbias) rCrDrX hidden_sizeintermediate_sizennLinear gate_projup_proj down_projr hidden_activationact_fnrGrXrHs r2rDzGemma3MLP.__init__s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV556 r1c|j|j|j||j|z}|SrK)rcrerarb)rGxrcs r2rLzGemma3MLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r1)r)r*r+r#rDrLrTrUs@r2rWrWs7/7r1rWc<eZdZddedeffd ZdZdZdZxZ S) Gemma3RMSNormdimepsct|||_tjt j ||_yrK)rCrDrlr_ Parameterr-zerosrN)rGrkrlrHs r2rDzGemma3RMSNorm.__init__s. ll5;;s#34 r1c|tj|jdjdd|jzzS)NrT)keepdim)r-rsqrtpowmeanrl)rGrhs r2_normzGemma3RMSNorm._norms45;;quuQx}}R}>IJJJr1c|j|j}|d|jjzz}|j|S)NrP)rvrRrNtype_as)rGrhoutputs r2rLzGemma3RMSNorm.forwardsCAGGI&3!2!2!445~~a  r1c^t|jjd|jS)Nz, eps=)tuplerNshaperlrGs r2 extra_reprzGemma3RMSNorm.extra_reprs' ))*+6$((<) r)r*r+rQrRrDrvrLr~rTrUs@r2rjrjs&5C5e5 K!=r1rjc^eZdZddeffd Zej edZxZ S)Gemma3RotaryEmbeddingrXct|t|drG|j;|jj d|jj d|_nd|_|j |_|j |_||_ t|j |_ |j|j|\}|_ |jd|d|j|_y)N rope_scaling rope_typetypedefaultinv_freqFrA)rCrDhasattrrgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrXr rope_init_fnattention_scalingrEroriginal_inv_freq)rGrXdevicerrHs r2rDzGemma3RotaryEmbedding.__init__s  6> *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.rK) r)r*r+r#rDr-no_gradrrLrTrUs@r2rrs4///"U]]_ <