UhTddlmZddlmZmZmZmZmZddlZddl m Z ddl m Z ddl mZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0ddl1m2Z2e*r ddl3m4Z4ddl5m6Z6e+jne8Z9eGdde'Z:e(de(Gdde"Z;edGdd e jxZ=Gd!d"e jxZ>Gd#d$e jxZ?d%Z@dHd&ZAd'ejd(eCd)ejfd*ZD dId+e jxd,ejd-ejd.ejd/eejd0eEd1eEfd2ZFGd3d4e jxZGGd5d6eZHe(Gd7d8e;ZIGd9d:e jxZJGd;dd?e;eZLGd@dAe jxZMe(GdBdCe;ZNe(dDGdEdFe;e2ZOgdGZPy)J) dataclass)CallableListOptionalTupleUnionN)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargs ModelOutputauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) AutoModel) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixin) BlockMask)make_flex_block_causal_maskceZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee ejdfed<dZ ee ejdfed<dZeejed <dZejed <dZee e ejed <dZee ejdfed <dZee ejdfed <dZeejed<y)CsmOutputWithPastaf Base class for the model autoregressive outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r)rtorch FloatTensor__annotations__r*r+rr,r-r.r/r0r1r2r3v/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/csm/modeling_csm.pyr(r(6s(*X)-D(5$$ %, $FE  $AEOXeE%*;*;$<=>E=AM8E%"3"3S"89:A:>Ju00#567>6:!2!23:.2%++2OS!8E%8I8I2J,K#LSKO%0A0A30F*G!HOHLhuU->->-C'DEL15M8E--.5r<r(z[ The bare Csm Model outputting raw hidden-states without any specific head on top. ) custom_introcBeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZdZy)CsmPreTrainedModelmodelTCsmDecoderLayerr+c6|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyt|trJ|j}t|dz D],}|j j |jd|.yt|tr&|j j j!dyy)N)meanstdr!g?)configinitializer_range isinstancennLinearweightdatanormal_biaszero_ Embedding padding_idxCsmCodebooksHead num_codebooksrange CsmRMSNormfill_)selfmodulerFrTis r= _init_weightsz CsmPreTrainedModel._init_weightss8kk++ fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  0 1"00M=1,- A ""1%--3C-@ A  + MM   $ $S ),r<N)r4r5r6r" config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_attention_backendr[r;r<r=r@r@qsQL&*#*+#4"5!N! $!"&*r<r@RMSNormc,eZdZdfd ZdZdZxZS)rVct|tjt j ||_||_y)z9 CsmRMSNorm is equivalent to T5LayerNorm N)super__init__rJ Parameterr8onesrLvariance_epsilon)rX hidden_sizeeps __class__s r=rkzCsmRMSNorm.__init__s1 ll5::k#:; #r<c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetor8float32powrErsqrtrnrL)rXr, input_dtypevariances r=forwardzCsmRMSNorm.forwardsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r<c^t|jjd|jS)Nz, eps=)tuplerLshapernrXs r= extra_reprzCsmRMSNorm.extra_reprs* ))*+6$2G2G1HIIr<)gư>)r4r5r6rkr|r __classcell__rqs@r=rVrVs$;Jr<rVc^eZdZddeffd Zej edZxZ S)CsmRotaryEmbeddingrGct|t|drG|j;|jj d|jj d|_nd|_|j |_|j |_||_ t|j |_ |j|j|\}|_ |jd|d|j|_y)N rope_scaling rope_typetypedefaultinv_freqF persistent)rjrkhasattrrgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrGr rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)rXrGdevicerrqs r=rkzCsmRotaryEmbedding.__init__s  6> *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.N) r4r5r6r"rkr8no_gradrr|rrs@r=rrs3/y/"U]]_ <