Uh}dZddlZddlmZmZmZmZddlZddlm cm Z ddl Zddlm Z ddl mZddlmZddlmZdd lmZmZdd lmZdd lmZmZd d lmZej:eZGdde j@Z!Gdde j@Z"Gdde j@Z#Gdde j@Z$Gdde j@Z%Gdde j@Z&Gdde j@Z'Gdde j@Z(Gdde j@Z)Gd d!e j@Z*Gd"d#e j@Z+eGd$d%eZ,eGd&d'e,Z-ed()Gd*d+e,eZ.gd,Z/y)-zPyTorch CPMAntN)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)auto_docstringlogging) CpmAntConfigcHeZdZdZdeffd ZdejfdZxZ S)CpmAntLayerNormzv We use Root Mean Square (RMS) Layer Normalization, please see https://arxiv.org/abs/1910.07467 for details." configct||j|_|j|_t j tj|j|_ yN) super__init__eps hidden_sizedim_normr Parametertorchemptyweightselfr __class__s |/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/cpmant/modeling_cpmant.pyrzCpmAntLayerNorm.__init__*sE ::** ll5;;v/A/A#BC hidden_statescp|jd|jk7r td|j}|j t j jdjdd}|t j||jzzj ||jz}|S)f Args: hidden_states (`torch.Tensor` of shape `(batch, seq_len, dim_in)`) z'hidden_states.size(-1) != self.dim_normT)dimkeepdim) sizerAssertionErrordtypetorfloat32powmeanrsqrtrr )r"r& old_dtypevariances r$forwardzCpmAntLayerNorm.forward1s   b !T]] 2 !JK K!''  ##EMM266q9>>2t>T&X5H)IIMMiX[_[f[ff r%) __name__ __module__ __qualname____doc__rrrTensorr7 __classcell__r#s@r$rr%s&D|D U\\ r%rceZdZdeffd Z d dej dej dejdej dee dee ej ej fd ee fd Z xZ S) CpmAntAttentionrcHt||j|_|j|_|j |_tj|j|j |j zd|_ tj|j|j |j zd|_ tj|j|j |j zd|_ tj|j |j z|jd|_ tjjd|_|j 0tjj#|j |_yd|_y)NFbiasr)r+)p)rrr dim_modelnum_attention_heads num_headsdim_headrLinear project_q project_k project_v attention_outrSoftmaxsoftmax dropout_pDropoutdropoutr!s r$rzCpmAntAttention.__init__?s ++33 4>>4>>DMM3QX]^4>>4>>DMM3QX]^4>>4>>DMM3QX]^YYt~~ 'Et~~\abxx''B'/    ' 88++f.>.>+?DLDLr%hidden_q hidden_kvattention_mask position_biasoutput_attentionspast_key_values use_cachec |jd}|jd} |jd} |j|} |j|} |j|} | j || |j |j jdddd} | j || |j |j jdddd} | j || |j |j jdddd} |Itj|d| gd} tj|d| gd} | jd} tj| | jddtj|j z }||z}tj||j |d| | tjd k(tj t#d |j$|j& }|j)|}tj||j |d| | tjd k(tj d|j$|j& }|r|}nd}|j*|j+|}tj|| }|j ||j | |j jdddd}|j-j || |j |j z}|j/|}d}|r| | f}|||fS) a Args: hidden_q (`torch.Tensor`): Input of transformer block(self-attention block). It can be the raw embedding of a batch of sequences. hidden_kv (`torch.Tensor` of shape `(batch, len_k, dim_model)`)): Tensor *key_value* and *query* of shape `(batch, len_k, dim_model)` attention_mask (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Avoid invalid areas to participate in the calculation of self-attention. position_bias (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Provide positional information to self-attention block. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. past_key_values (`Tuple[torch.Tensor, torch.Tensor]`, *optional*): Cached past key and value projection states. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). rrr*r NrDr)Fz-inf)devicer/)r-rKrLrMviewrHrIpermutercatmatmul transposemathsqrt masked_filltensor scalar_tensorfloatr]r/rPrS contiguousrN)r"rTrUrVrWrXrYrZ batch_sizelen_qlen_kquerykeyvaluescore attn_weightss r$r7zCpmAntAttention.forwardRs8]]1%  a q!x(nnY'y) :udnndmmLTTUVXY[\^_`hhz5$..$--HPPQRTUWXZ[\ :udnndmmLTTUVXY[\^_`  &))_Q/52>CIIq159rBEHHRLE UCMM"b$9:TYYt}}=UU %!!     Aue < U@S S   f ell%++ V   U#!!     Aue < U@S S   %,,ekk J   LL << #LL'E UE* :t~~udmmLTTUVXY[\^_`  "'' E4>>DMM;YZ""5) "ElOlO33r%)FNN) r8r9r:rrrr< BoolTensorrboolrr7r=r>s@r$r@r@>s | 2-2GK$(Q4,,Q4<<Q4(( Q4 || Q4 $D> Q4"% ell(B"CDQ4D>Q4r%r@ceZdZdeffd Z d dej dej deej deedee ej ej fdeef d Z xZ S) CpmAntSelfAttentionBlockrct|t||_t ||_|j r/tjj|j |_ yd|_ yr) rrrlayernorm_before_attentionr@self_attentionrQrrrRrSr!s r$rz!CpmAntSelfAttentionBlock.__init__sT *9&*A'-f5    88++F,<,<=DLDLr%r&rVrWrXrYrZc |j|}|j|||||||}|\}}} |j|j|}||z}||| fS)a Args: hidden_states (`torch.Tensor` of shape `(batch, len_seq, dim_model)`): Input of transformer block(self-attention block). It can be the raw embedding of a batch of sequences. attention_mask (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Avoid invalid areas to participate in the calculation of self-attention. position_bias (`torch.Tensor` of shape `(batch, len_seq, len_seq)`): Provide positional information to self-attention block. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. past_key_values (`Tuple(torch.FloatTensor)`, *optional*): Cached past key and value projection states. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )rwrxrS) r"r&rVrWrXrYrZoutputsrqcurrent_key_values r$r7z CpmAntSelfAttentionBlock.forwardsu211-@%% Wnm=NP_aj 4;00 << #ll7+G%/ l,===r%NFNN r8r9r:rrrr<rrsrr7r=r>s@r$rurus | 15,1GK$($>||$> $>  - $> $D> $> "% ell(B"CD $>D>$>r%rucDeZdZdeffd Zdej fdZxZS)CpmAntDenseGatedACTrc,t|tj|j|j d|_tj|j|j d|_tjj|_ yNFrB) rrrrJrdim_ffw_0w_1rGELUactr!s r$rzCpmAntDenseGatedACT.__init__s[ 99V//UK99V//UK88==?r%r&cr|j|j|}|j|}||z}|S)zTransform an input tensor from one feature space to another via a nonlinear operation Args: hidden_states (`torch.Tensor` of shape `(batch, seq_len, dim_in)`) )rrr)r"r& gate_scores r$r7zCpmAntDenseGatedACT.forwards9 XXdhh}56 / "]2 r% r8r9r:rrrr<r7r=r>s@r$rrs#|# U\\ r%rcDeZdZdeffd Zdej fdZxZS)CpmAntFeedForwardrc(t|t||_|j/t j j|j|_nd|_t j|j|jd|_ yr) rrrw_inrQrrrRrSrJrrw_outr!s r$rzCpmAntFeedForward.__init__sg '/    ' 88++F,<,<=DLDLYYv}}f.@.@uM r%r&c|j|}|j|j|}|j|}|S)r()rrSrr"r&s r$r7zCpmAntFeedForward.forwards>  -0 << # LL7M =1 r%rr>s@r$rrs!N|N U\\ r%rcDeZdZdeffd Zdej fdZxZS)CpmAntFFNBlockrct|t||_t ||_|j r/tjj|j |_ yd|_ yr) rrrlayernorm_before_ffnrffnrQrrrRrSr!s r$rzCpmAntFFNBlock.__init__sS $3F$;!$V,    88++F,<,<=DLDLr%r&c|j|}|j|}|j|j|}||z}|S)z Args: hidden_states (`torch.Tensor` of shape `(batch, len_seq, dim_model)`): Hidden states before feed forward layer. )rrrS)r"r& ln_outputsrzs r$r7zCpmAntFFNBlock.forwardsJ..}= ((:& << #ll7+G%/ r%rr>s@r$rrs  | ||r%rceZdZdeffd Z d dej dej deej deedee ej ej fdeef d Z xZ S) CpmAntTransformerBlockrcbt|t||_t ||_yr)rrruself_attrrr!s r$rzCpmAntTransformerBlock.__init__!s& 08 !&)r%r&rVrWrXrYrZch|j||||||}|\}}}|j|}|||fS)a Args: hidden_states (`torch.Tensor`): Input to the layer of shape `(batch, seq_len, dim_model)` attention_mask (`torch.Tensor`): Avoid invalid areas to participate in the calculation of shape `(batch, seq_len, seq_len)` position_bias (`torch.Tensor`): Provides position information to attention mechanism of shape `(num_heads, seq_len, seq_len)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. past_key_values (`Tuple[torch.Tensor, torch.Tensor])`, *optional*): Cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )rVrWrXrYrZ)rr) r"r&rVrWrXrYrZrqr{s r$r7zCpmAntTransformerBlock.forward&sU2 )'/+ & :G6 |%6/ l,===r%r|r}r>s@r$rr s*|*15,1GK$(&>||&> &>  - &> $D> &> "% ell(B"CD &>D>&>r%rceZdZdeffd Z d dej dej dej deedeedee ej ej fd eefd Z xZ S) CpmAntEncoderrct||j|_t j t |jDcgc] }t|c}|_t||_ ycc}wr) rrnum_hidden_layers num_layersr ModuleListrangerlayersroutput_layernorm)r"rithr#s r$rzCpmAntEncoder.__init__Ps[  22mmuUYUdUdOe$f%;F%C$fg / 7%gsA6r&rVrWrXoutput_hidden_statesrYrZc |rdnd}|rdnd} |rdnd} t|jD]9\} } |r||fz }| |||||r|| nd|} | \}}}|r| |fz } |4| |fz} ;|j|}|r||fz }|| || fS)a% Args: hidden_states (`torch.Tensor`): Input to the layer of shape `(batch, seq_len, dim_model)` attention_mask (`torch.Tensor`): Avoid invalid areas to participate in the calculation of shape `(batch, seq_len, seq_len)` position_bias (`torch.Tensor`): Provides position information to attention mechanism of shape `(num_heads, seq_len, seq_len)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. past_key_values (`Tuple[torch.Tensor, torch.Tensor])`, *optional*): Cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). N)rXrYrZ) enumeraterr)r"r&rVrWrXrrYrZall_hidden_statesall_self_attnscurrent_key_valuesilayer layer_outputsrqr{s r$r7zCpmAntEncoder.forwardWs8#7BD0d#,R$!$++. OHAu#!m%55!!"36E 24# M>K :M<): s@r$rrOs8|8-1/3GK$(6T||6T 6T|| 6T $D> 6T 'tn 6T"% ell(B"CD6TD>6Tr%rcVeZdZfdZdej dej fdZxZS)CpmAntIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) rrrrJrintermediate_sizedense isinstance hidden_actstrr intermediate_act_fnr!s r$rzCpmAntIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r%r&returncJ|j|}|j|}|Sr)rrrs r$r7zCpmAntIntermediate.forwards& =1 00? r%r8r9r:rrr<r7r=r>s@r$rrs#9U\\ellr%rceZdZdeffd Zdej dej dej dej fdZdZd d Z xZ S) CpmAntSegmentPositionEmbeddingrcbt||j|_|j|_|j |_|j|_ tjtj|j|jz|jz|j|_yr)rrrGrHposition_bias_num_buckets num_bucketsposition_bias_max_distance max_distance segment_types num_segmentsrrrrrelative_attention_biasr!s r$rz'CpmAntSegmentPositionEmbedding.__init__s 33!;;"=="00')|| KK$$v';';;f>^>^^** ( $r%key_pos query_pos key_segment query_segmentc 0tj5|jd}|jd}|jd}|jd|jdk7r0td|jdd|jdd||jdk7s||jdk7r!td|d|jdd||jdk7r!td|d|jdd|j |d|}|j ||d}|j |d|}|j ||d}|j ||}||j z}|jtj|tj|j dddftj|tj|j dddfz |j |j } tj||k(| dddddf|}dddtj|j} | j!dd dd j#} | S#1swYMxYw) Nrrz>key_pos.size(0) should be equal to query_pos.size(0), but got z and !z7keylen should be equal to key_segment.size(1), but got z;querylen should be equal to query_segment.size(1), but got r)r/r])rrr r*)rno_gradr-r.r^!_segment_relative_position_bucketr_position_bucketarangeint32r]rwhereF embeddingrr_ri) r"rrrrbatchkeylenquerylenrelative_position_bucketabsolute_position_bucketembedss r$r7z&CpmAntSegmentPositionEmbedding.forwards]]_% LLOE\\!_F ~~a(H||A).."33$TU\UaUabcUdTeejktkykyz{k|j}}~))!,,M>r%c.d}|dz}|dkDjtj|z}tj|}|dz}||k}|tj|j |z t j||z z ||z zjtjz}tj|tj||dz }|tj||jtj|z }|S)Nrr*r) r0rrabslogrhrcmin full_liker)r"relative_positionrrrelative_buckets max_exactis_smallrelative_postion_if_larges r$rz/CpmAntSegmentPositionEmbedding._position_buckets -155ekkB[P!II&781$ $y0$- II'--/); <hh|i/0 1Y& ( "U[[/ %! %*II % OO5{Q G% ! EKK2C2F2Fu{{2SUnoor%) ) r8r9r:rrrr<r7rrr=r>s@r$rrsU  |  22<<2\\ 2 || 2h? r%rcneZdZfdZdej dej dej fdZxZS) CpmAntOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ y)N)r) rrrrJrrr LayerNormlayer_norm_epsrRhidden_dropout_probrSr!s r$rzCpmAntOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r%r& input_tensorrcr|j|}|j|}|j||z}|Sr)rrSr)r"r&rs r$r7zCpmAntOutput.forwards7 =1  ]3 }|'CD r%rr>s@r$rrs1> U\\RWR^R^r%rceZdZeZdZdZy)CpmAntPreTrainedModelcpmantct|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyt|tr&|jjjdyt|tr<|j jj d|j jyy)zInitialize the weightsg)r3stdNg?)rrrJr datanormal_rinit_stdrCzero_ Embedding padding_idxrfill_rrr)r"modules r$ _init_weightsz#CpmAntPreTrainedModel._init_weightss[ fbii ( MM   & &CT[[5I5I & J{{&   &&('  - MM   & &CT[[5I5I & J!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S )  0 MM   $ $S )  > ?  * * / / 7 7SdkkFZFZ 7 [@r%N)r8r9r:r config_classbase_model_prefixrrr%r$rr sL \r%rceZdZdeffd ZdZdZdZe dde e jde e de e d e e e e jd e e d e e d ee e jeffd ZxZS) CpmAntModelrct||t||_t j |j |j|_t j |j|j|jzz|j|_ t||_|j|_ |j|_ |jyr)rrrencoderrrrrsegment_embedding vocab_size prompt_types prompt_lengthinput_embeddingrrW post_initr!s r$rzCpmAntModel.__init__%s  $V, !#f.B.BFDVDV!W!||    3 3f6J6J J JFL^L^ z&CpmAntModel.forward..sefers)last_hidden_staterYr& attentions)rrXruse_return_dictrZr/rrr0r]rsumr`rrrrr-zerosfulltuplerrrir rrrWr )r"rrXrrYrZr rr/r]segmentrr seq_lengthrpositionr past_lengthr&segment_statesrVrWpresent_key_valuesrall_attentionsnew_attentions attentionnew_hidden_states hidden_states r$r7zCpmAntModel.forwardJs.(2C1N-TXT_T_TqTq$8$D $++JjJj &1%WWM55iwPVW**8XwP ';<(:;%aKL!&;< %aq&89 OS||      P L )+s@r$rr#s | $*$-1,0/3@D$(&*g ELL)g $D>g 'tn g "%ell(;"<= g D> g d^g  uU\\"$;; <g g r%rzy The CPMAnt Model with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc>eZdZdgZdeffd Ze ddeejdee e ejejfdee dee dee d eejd ee d eejd e e effd ZdZdZdZdZdZxZS)CpmAntForCausalLMzlm_head.weightrct||t||_t j |j |j|j|jzzd|_ |jyr) rrrrrrJrrrrlm_headr r!s r$rzCpmAntForCausalLM.__init__sd  !&) yy    1 1F4G4G&J^J^4^ ^ej   r%rrYrZrXrlabelsr rVrc  ||n|jj}|j||||||} |r | jn| d} |j | } d} |At }|| j d| jd|j d} |s| f| ddz}| | f|zS|St| | | j| j| jS)u< input_ids (`torch.Tensor` of shape `(batch_size, seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`CPMAntTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Example: Text Generation with CpmAntForCausalLM. ```python >>> from transformers import CPMAntTokenizer, CpmAntForCausalLM >>> texts = "今天天气不错，" >>> model = CpmAntForCausalLM.from_pretrained("openbmb/cpm-ant-10b") >>> tokenizer = CPMAntTokenizer.from_pretrained("openbmb/cpm-ant-10b") >>> input_ids = tokenizer(texts, return_tensors="pt") >>> outputs = model.generate(**input_ids) >>> output_texts = tokenizer.batch_decode(outputs) >>> print(output_texts) ['今天天气不错，阳光明媚，我和妈妈一起去超市买东西。\n在超市里，我看到了一个很好玩的玩具，它的名字叫“机器人”。它有一个圆圆的脑袋，两只圆圆的眼睛，还有一个圆圆的'] ``` Nrr)r)losslogitsrYr&r') rr(rr&r<rr^r-r rYr&r')r"rrYrZrXrr=r rVr model_outputr&r@r? loss_funcoutputs r$r7zCpmAntForCausalLM.forwardsP&1%QZ\g ;F 66O!0!3H!=OA !0!3H!=OA  > as<)NNNNNNNN)r8r9r:_tied_weights_keysrrrrrr<rrrsrr r7rrrIrLrQr=r>s@r$r:r:s  ++|-1MQ$(,0/3)-&*15? ELL)? "$uU\\5<<-G'H"IJ? D> ? $D> ? 'tn ? &? d^? !.?  u,, -? ? B+1&r%r:)r:rr)0r;rctypingrrrrrtorch.nn.functionalr functionalrtorch.utils.checkpointtorch.nnr activationsr generationr modeling_outputsr r modeling_utilsrutilsrrconfiguration_cpmantr get_loggerr8loggerModulerr@rurrrrrrrrrrr:__all__rr%r$rbsx // %!)O-,.   H %bii2e4biie4P.>ryy.>b"))( 4RYY6,>RYY,>^>TBII>TD  Y RYYY z 299 \O\\.N 'N N b `-`  `F Hr%