UhddlmZmZddlZddlmZddlmZddlmZddl m Z dd l m Z m Z mZmZd d lmZe j$eZGd d ej*ZGdde ZGddeZGddeZGdde ZgdZy))OptionalTupleN)nn)ACT2FN)Cache)logging)GraniteMoeDecoderLayerGraniteMoeForCausalLMGraniteMoeModelGraniteMoePreTrainedModel)GraniteMoeSharedConfigc`eZdZdZdeffd ZdejdejfdZxZ S)GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configchtt| |j|_|j |_t |j|_tj|j|jdzd|_ tj|j|jd|_ y)Nr F)bias) superr__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr __class__s /var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/granitemoeshared/modular_granitemoeshared.pyrzGraniteMoeSharedMLP.__init__-s !413 ,,!:: !2!23IIdoot7G7G!7KRWXYYt'7'7uU hidden_statesreturnc|j|}|jdd}|j|d|dz}|j|}|S)Nr )dimrr)rchunkrr)r!r%chunked_hidden_statess r#forwardzGraniteMoeSharedMLP.forward6s^))-8 - 3 3A2 3 >(=a(@ADYZ[D\\ **=9 r$) __name__ __module__ __qualname____doc__rrtorchTensorr, __classcell__r"s@r#rr$s2V5VU\\ellr$rcreZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee d ee ejejfd e ejee ejejfffdZxZS)GraniteMoeSharedDecoderLayerr layer_idxctt||||jdk(rd|_yt||_y)Nr)rrrr shared_mlpr!rr7r"s r#rz%GraniteMoeSharedDecoderLayer.__init__?s3 +"("A"AQ"F$L_`fLgr$r%attention_mask position_idspast_key_valueoutput_attentions use_cachecache_positionoutput_router_logitsposition_embeddingsr&c ~|} |j|}|jd|||||||| d| \}} } | ||jzz}|} |j|}|j |\}}|j |}n||j |z}~| ||jzz}|f}|r|| fz }|r|| fz }|r||fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r%r;r<r=r>r?r@rB)input_layernorm self_attnresidual_multiplierpost_attention_layernormblock_sparse_moer9)r!r%r;r<r=r>r?r@rArBkwargsresidualself_attn_weightspresent_key_valuemoe_hidden_states router_logitsoutputss r#r,z$GraniteMoeSharedDecoderLayer.forwardCsL!,,]; ?Mdnn ? ')%)/) 3 ?  ? ; (*;!=43K3K#KK !55mD +/+@+@+O(= ?? "-M- 0NNM  =43K3K#KK  "  )+ +G  )+ +G   ' 'Gr$)NNNFFNFN)r-r.r/rintrr1r2r LongTensorrboolr FloatTensorr,r3r4s@r#r6r6>sh5h#h2637*.,1$)59/4KOR||R!.Ru//0 R ! R $D> RD>R!!1!12R'tnR&eELL%,,,F&GHR u  (51B1BEDUDU1U+V"WW XRr$r6ceZdZeZdgZy)GraniteMoeSharedPreTrainedModelr6N)r-r.r/r config_class_no_split_modulesrDr$r#rVrVs)L78r$rVc$eZdZdeffd ZxZS)GraniteMoeSharedModelrc t||tjt |j Dcgc]}t ||c}|_ycc}wN)rrr ModuleListrangenum_hidden_layersr6layersr:s r#rzGraniteMoeSharedModel.__init__sE  mmNSTZTlTlNm n )&) < n  nsA)r-r.r/rrr3r4s@r#rZrZs 5  r$rZc*eZdZdgZdeffd ZxZS)GraniteMoeSharedForCausalLMzlm_head.weightrcdt||t||_|j yr\)rrrZmodel post_initr s r#rz$GraniteMoeSharedForCausalLM.__init__s&  *62  r$)r-r.r/_tied_weights_keysrrr3r4s@r#rbrbs*+5r$rb)rbrZrV)typingrrr1r activationsr cache_utilsrutilsr granitemoe.modeling_granitemoer r r rconfiguration_granitemoesharedr get_loggerr-loggerModulerr6rVrZrb__all__rDr$r#rqs # !  C   H %"))4W#9Wt9&?9  O "7 fr$