Uh7^dZddlZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl mZddl mZddlmZmZmZdd lmZdd lmZdd lmZmZdd lmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1e/jde3Z4dZ5GddejlZ7GddejlZ8Gdde8Z9GddejlZ:e8e9dZ;GddejlZ<GddejlZ=Gd d!ejlZ>Gd"d#ejlZ?Gd$d%ejlZ@Gd&d'ejlZAGd(d)ejlZBGd*d+ejlZCGd,d-ejlZDGd.d/ejlZEGd0d1ejlZFe-Gd2d3e&ZGeGd4d5e,ZHe-d67Gd8d9eGZIe-d:7Gd;dd?eGeZKe-Gd@dAeGZLe-dB7GdCdDeGZMe-dE7GdFdGeGZNe-GdHdIeGZOe-GdJdKeGZPe-GdLdMeGZQgdNZRy)OzPyTorch BERT model.N) dataclass)ListOptionalTupleUnion)version)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GenerationMixin)#_prepare_4d_attention_mask_for_sdpa*_prepare_4d_causal_attention_mask_for_sdpa) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringget_torch_versionlogging) BertConfigc L ddl}ddl}ddl}t jj|}tjd||jj|}g}g} |D]^\} } tjd| d| |jj|| } |j| | j| `t|| D]\} } | jd} t!d| Dr(tjd dj#| R|} | D]}|j%d |r|jd |}n|g}|dd k(s|dd k(r t'| d} nW|ddk(s|ddk(r t'| d} n:|ddk(r t'| d} n%|ddk(r t'| d} n t'| |d} t+|dk\st-|d}| |} dddk(r t'| d} n|d k(r|j/| } | j0| j0k7r&t3d| j0d| j0d tjd| t7j8| | _|S#t$rtj dwxYw#t($r+tjd dj#| YwxYw#t2$r1}|xj4| j0| j0fz c_d}~wwxYw)z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape /c3$K|]}|dv yw))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns x/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/bert/modeling_bert.py z*load_tf_weights_in_bert..Rs  n n z Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierr#i _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzipsplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshape ValueErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr>nptftf_path init_varsnamesarraysnamerVarraypointerm_name scope_namesnumes r0load_tf_weights_in_bertrl5s, ggoo01G KK8 BC''0I E F  e (l5'BC&&w5 T e  5&),/ ezz#     KK)CHHTN#34 5  'F||,f5 hhy&9 %h 1~)[^w-F!'84Q=0KNf4L!'62Q#33!'84Q7*!'<8%g{1~>G;1$+a.)!#,+ ', #$<= (gx0G x LL'E }} + >'--@QRWR]R]Q^^i!jkk,  078''. Y,/Z L   Q   Z&KK)CHHTN+; <=  FFw}}ekk2 2F  s5 J9J2?K) J/20K&%K&) L#2,LL#c eZdZdZfdZ d deejdeejdeejdeejde dejf d Z xZ S) BertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c>t|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_t#|dd|_|j'dt)j*|jj-dd|j'd t)j.|j0j3t(j4 dy) N) padding_idxepsposition_embedding_typeabsolute position_ids)r#F) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutrQrsregister_bufferrYarangeexpandzerosrusizelongselfr] __class__s r0r|zBertEmbeddings.__init__s/ !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  input_idsrxru inputs_embedspast_key_values_lengthreturncZ||j}n|jdd}|d}||jdd|||zf}|st|dr-|jddd|f}|j |d|} | }n:t j |t j|jj}||j|}|j|} || z} |jdk(r|j|} | | z } |j| } |j| } | S)Nrvr#rxrrzdevicert)rruhasattrrxrrYrrrrrrsrrr) rrrxrurr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrs r0forwardzBertEmbeddings.forwardsH  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !t-.*.*=*=a*n*M'3J3Q3QR]^_R`bl3m0!A!&[ SWSdSdSkSk!l   00;M $ : :> J"%::  ' ': 5"&":":<"H  - -J^^J/ \\*- r)NNNNr) __name__ __module__ __qualname____doc__r|rrY LongTensor FloatTensorrTTensorr __classcell__rs@r0rnrn~sQ *15593759&' 'E,,-'!!1!12'u//0 '   1 12 ' !$ ' 'rrncPeZdZd fd Zdej dej fdZ ddej deejdeejdeejd eejd ee e ejd ee de ej fd Z xZ S)BertSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_|xs t#|dd|_|j$dk(s|j$d k(rF|j&|_tj(d |j&zd z |j|_|j,|_y) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rsrt relative_keyrelative_key_queryr<r#)r{r|rnum_attention_headsrrWrTattention_head_size all_head_sizer Linearquerykeyvaluerattention_probs_dropout_probrrQrsrr}distance_embedding is_decoderrr]rsrs r0r|zBertSelfAttention.__init__s     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++rxrc|jdd|j|jfz}|j|}|j ddddS)Nrvrr<r#r )rrrviewpermute)rr new_x_shapes r0transpose_for_scoresz&BertSelfAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FF; yyAq!$$r hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsc$|j|}|du} | r||d} |d} |}n| rC|j|j|} |j|j|} |}n|y|j|j|} |j|j|} t j |d| gd} t j |d| gd} n@|j|j|} |j|j|} |j|} |du} |j r| | f}t j| | jdd}|jdk(s|jdk(r| jd| jd}}| rDt j|dz tj|j jdd}n@t j|tj|j jdd}t j|tj|j jdd}||z }|j!||j"zdz }|j%| j& }|jdk(rt j(d | |}||z}nE|jdk(r6t j(d | |}t j(d | |}||z|z}|t+j,|j.z }|||z}t0j2j5|d}|j7|}|||z}t j|| }|j9dddd j;}|j=dd|j>fz}|j|}|r||fn|f}|j r||fz}|S)Nrr#r<dimrvrrrryzbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrrrrYcatrmatmulrUrsrVtensorrrrrrrtorzeinsummathsqrtrr functionalsoftmaxrr contiguousrr)rrrrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r0rzBertSelfAttention.forwards!JJ}5 3$> ."<&q)I(+K3N 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB "$. ??(5N!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM] ?? 11GrNNNNNNF) rrrr|rYrrrrrboolrrrs@r0rrs,4%ell%u||%7;15=A>BDH,1c||c!!2!23cE--. c ((9(9: c !)):): ; c!uU->->'?!@Ac$D>c u|| crrceZdZd fd Z d dej deej deejdeejdeejdeeeejdee d eej ffd Z xZ S) BertSdpaSelfAttentionct||||j|_t j t t j dk|_y)Nrsz2.2.0)r{r|r dropout_probrparser!require_contiguous_qkvrs r0r|zBertSdpaSelfAttention.__init__EsH 9PQ"??&-mm4E4G&H7==Y`Ka&a#rrrrrrrrrc p|jdk7s|s|*tjdt||||||||S|j \}} } |j |j|} |du} | r|n|} | r|n|}| r*|r(|djd| jdk(r|\}}n||j |j| }|j |j| }|:| s8tj|d|gd}tj|d|gd}|jr||f}|jrK| jj dk(r2|0| j#} |j#}|j#}|jr | s || dkDrdnd }tj$j&j)| ||||j*r |j,nd | }|j/dd}|j1|| |j2}|f}|jr||fz}|S) NrtaBertSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support non-absolute `position_embedding_type` or `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rr<r#rcudaTF) attn_mask dropout_p is_causal)rsrB warning_oncer{rrrrrVrrrYrrrrtyperr rscaled_dot_product_attentiontrainingrrUreshaper)rrrrrrrrbsztgt_len_rrcurrent_statesrrr attn_outputrrs r0rzBertSdpaSelfAttention.forwardKs`  ' ': 59JiNc   H 7?%&! (,,.Wa// =0IJ 3$>2D.-3E/> .^A5F5L5LQ5OSaSgSghiSj5j%3 "I{11$((>2JKI33DJJ~4NOK)2D!II~a'8)&D!L #ii):K(HaP ??(5N  & &;+=+=+B+Bf+LQ_Qk%002K!,,.I%002KOO,>>CY^ehi^iDot hh))FF   $+/==d''c G "++Aq1 !))#w8J8JK . ?? 11Grrr) rrrr|rYrrrrrrrrs@r0rrDsb2615=A>BDH,1[||[!.[E--. [ ((9(9: [ !)):): ; [!uU->->'?!@A[$D>[ u|| [[rrcneZdZfdZdej dej dej fdZxZS)BertSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNrq) r{r|r rrdenserrrrrrs r0r|zBertSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rr input_tensorrcr|j|}|j|}|j||z}|Srr rrrrr s r0rzBertSelfOutput.forward7 =1  ]3 }|'CD rrrrr|rYrrrrs@r0rr1> U\\RWR^R^rr)eagersdpaceZdZd fd ZdZ d dej deejdeejdeejdeejdee e ejd ee d e ej fd Z xZ S) BertAttentionct|t|j|||_t ||_t|_y)Nr) r{r|BERT_SELF_ATTENTION_CLASSES_attn_implementationrroutputset pruned_headsrs r0r|zBertAttention.__init__sC /0K0KL ,C  %V, Erc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrr#r)rSrrrrrrrrrrr runion)rheadsindexs r0 prune_headszBertAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rrrrrrrrrc p|j|||||||}|j|d|} | f|ddz} | S)Nrr#)rr) rrrrrrrr self_outputsattention_outputrs r0rzBertAttention.forwardsWyy    ! "    ;;|A F#% QR(88rrr) rrrr|r!rYrrrrrrrrs@r0rrs";*7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| rrcVeZdZfdZdej dej fdZxZS)BertIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r{r|r rrintermediate_sizer  isinstance hidden_actstrrintermediate_act_fnrs r0r|zBertIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rrrcJ|j|}|j|}|Sr)r r,rrs r0rzBertIntermediate.forwards& =1 00? rrrs@r0r&r&s#9U\\ellrr&cneZdZfdZdej dej dej fdZxZS) BertOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr ) r{r|r rr(rr rrrrrrs r0r|zBertOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rrr rcr|j|}|j|}|j||z}|Srrrs r0rzBertOutput.forwardrrrrs@r0r0r0rrr0ceZdZfdZ d dej deejdeejdeejdeejdeeeejdee d eej fd Z d Z xZ S) BertLayercft||j|_d|_t ||_|j |_|j|_|jr,|j st|dt |d|_ t||_ t||_ y)Nr#z> should be used as a decoder model if cross attention is addedrtr)r{r|chunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrWcrossattentionr& intermediater0rrs r0r|zBertLayer.__init__s '-'E'E$&v. ++#)#=#=  # #?? D6)g!hii"/PZ"[D ,V4 ( rrrrrrrrrc ||ddnd}|j|||||} | d} |jr | dd} | d} n| dd} d} |jrT|Rt|dstd|d||d dnd}|j | ||||||}|d} | |ddz} |d}  | z} t |j |j|j| }|f| z} |jr|  fz} | S) Nr<)rrrr#rvr:z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) r8rrrWr:rfeed_forward_chunkr6r7)rrrrrrrrself_attn_past_key_valueself_attention_outputsr$rpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputs r0rzBertLayer.forwards}:H9S>"1#5Y] !%   /3 "0" 2!4 ??,Qr2G 6r : ,QR0G'+$ ??4@4!12 =dVDDD @N?Yrs(;_c %&*&9&9 %&)!' # 7q9  7" ==G,C2+F ( 14P P 0  # #T%A%A4CSCSUe  /G+ ??!2 44GrcL|j|}|j||}|Sr)r;r)rr$intermediate_outputrDs r0r=zBertLayer.feed_forward_chunk]s,"//0@A{{#68HI rr) rrrr|rYrrrrrrr=rrs@r0r4r4 s )"7;15=A>BDH,1?||?!!2!23?E--. ? ((9(9: ? !)):): ; ?!uU->->'?!@A?$D>? u|| ?Brr4cDeZdZfdZ ddej deejdeejdeejdeejdeeeejdee d ee d ee d ee d e eej e ffd Z xZ S) BertEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r{r|r]r ModuleListrangenum_hidden_layersr4layergradient_checkpointing)rr]rrs r0r|zBertEncoder.__init__dsN  ]]uVE]E]?^#_!If$5#_` &+#$`sA#rrrrrpast_key_valuesrroutput_hidden_states return_dictrc | rdnd} |rdnd} |r|jjrdnd} |jr%|jr|rtj dd}|rdnd}t |jD]\}}| r| |fz} |||nd}|||nd}|jr/|jr#|j|j|||||||}n||||||||}|d}|r ||dfz }|s|| |dfz} |jjs| |dfz} | r| |fz} | std||| | | fDSt||| | | S) Nr-zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrvr#r<c3$K|]}|| ywrr-)r.vs r0r1z&BertEncoder.forward..s = r2)last_hidden_staterOr attentionscross_attentions) r]r9rNrrBr enumeraterM_gradient_checkpointing_func__call__tupler)rrrrrrrOrrrPrQall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskr layer_outputss r0rzBertEncoder.forwardjs#7BD$5b4%64;;;Z;Zr`d  & &4==##p" #,R$(4# VOA|#$58H$H!.7.CilO3B3N_Q/TXN**t}} $ A A ))!"#)*"% ! !-!"#)*"%! *!,M"}R'8&::" &9]1=MBEI$(,1/4&*S ||S !!2!23S E--. S ((9(9: S !)):): ; S "%e.?.?(@"ABS D>S $D>S 'tnS d^S  uU\\"$MM NS rrHcVeZdZfdZdej dej fdZxZS) BertPoolerct|tj|j|j|_tj |_yr)r{r|r rrr Tanh activationrs r0r|zBertPooler.__init__s9 YYv1163E3EF '')rrrc\|dddf}|j|}|j|}|S)Nr)r rh)rrfirst_token_tensor pooled_outputs r0rzBertPooler.forwards6+1a40 #56  6 rrrs@r0reres#$ U\\ellrrecVeZdZfdZdej dej fdZxZS)BertPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr ) r{r|r rrr r)r*r+rtransform_act_fnrrrs r0r|z$BertPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrrrcl|j|}|j|}|j|}|Sr)r rorr.s r0rz#BertPredictionHeadTransform.forwards4 =1 --m< }5 rrrs@r0rmrms$UU\\ellrrmc*eZdZfdZdZdZxZS)BertLMPredictionHeadcHt|t||_t j |j |jd|_t jtj|j|_ |j|j_ y)NF)r8) r{r|rm transformr rrr~decoder ParameterrYrr8rs r0r|zBertLMPredictionHead.__init__sm 4V<yy!3!3V5F5FUS LLV->->!?@ !II rc:|j|j_yr)r8rurs r0 _tie_weightsz!BertLMPredictionHead._tie_weightss II rcJ|j|}|j|}|Sr)rtrur.s r0rzBertLMPredictionHead.forwards$}5  ]3 r)rrrr|ryrrrs@r0rrrrs &&rrrcVeZdZfdZdej dej fdZxZS)BertOnlyMLMHeadcBt|t||_yr)r{r|rr predictionsrs r0r|zBertOnlyMLMHead.__init__s /7rsequence_outputrc(|j|}|Sr)r~)rrprediction_scoress r0rzBertOnlyMLMHead.forwards ,,_=  rrrs@r0r|r|s#8!u||! !rr|c$eZdZfdZdZxZS)BertOnlyNSPHeadclt|tj|jd|_yNr<)r{r|r rrseq_relationshiprs r0r|zBertOnlyNSPHead.__init__s'  " &*<*.  - KK   " " $ MM   $ $S )  4 5 KK   " " $6rN) rrrr$ config_classrlload_tf_weightsbase_model_prefixsupports_gradient_checkpointing_supports_sdparr-rr0rrs"L-O&*#N%rrceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<y)BertForPreTrainingOutputa\ Output type of [`BertForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossprediction_logitsseq_relationship_logitsrrV)rrrrrrrYr__annotations__rrrrrVr-rr0rr2s~2)-D(5$$ %,59x 1 129;?Xe&7&78?8#> '-'E'E$ rc.|jjSrrrrxs r0get_input_embeddingszBertModel.get_input_embeddingsvs...rc&||j_yrr)rrs r0set_input_embeddingszBertModel.set_input_embeddingsys*/'rc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrMr8r!)rheads_to_prunerMrs r0 _prune_headszBertModel._prune_heads|sE +002 CLE5 LL  u % / / ; ;E B CrrrrxrurrrrrOrrrPrQrc | | n|jj} | | n|jj} | | n|jj} |jjr| | n|jj } nd} | | t d|#|j|||j}n!||jdd}n t d|\}}| |jn |j}| | ddjdnd}|pt|jdr4|jjddd|f}|j||}|}n&tj |tj"|}|j||||| }|tj$|||zf| }|j&d k(xr|j(d k(xr |duxr| }|rQ|j+dk(r>|jjrt-||||}n+t/||j0| }n|j3||}|jjrs|q|j\}}}||f}|tj$|| }|r,|j+dk(rt/||j0| }n|j5|}nd}|j7||jj8}|j;|||||| | | | |  }|d}|j<|j=|nd}| s ||f|ddzSt?|||j@|jB|jD|jFS)NFzDYou cannot specify both input_ids and inputs_embeds at the same timervz5You have to specify either input_ids or inputs_embedsrr<rxr)rrurxrr)rrrt)r) rrrrrOrrrPrQr#)rU pooler_outputrOrrVrW)$r]rrPuse_return_dictrrrW%warn_if_padding_and_no_attention_maskrrrVrrrxrrYrronesrrsrrrrzget_extended_attention_maskinvert_attention_mask get_head_maskrLrrrrOrrVrW) rrrrxrurrrrrOrrrPrQr batch_sizerrrrrembedding_outputuse_sdpa_attention_masksextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskencoder_outputsrrks r0rzBertModel.forwards"2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@TDSC^!3A!6!L $D>L 'tnL d^L  uU\\"$PP QL L rrz Bert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. ceZdZddgZfdZdZdZe ddee jdee jdee jd ee jd ee jd ee jd ee jd ee jdee dee dee de e e jeffdZxZS)BertForPreTrainingpredictions.decoder.biascls.predictions.decoder.weightct||t||_t ||_|j yr)r{r|rrrclsrrs r0r|zBertForPreTraining.__init__s4  f% '/ rcB|jjjSrrr~rurxs r0get_output_embeddingsz(BertForPreTraining.get_output_embeddings&xx##+++rc||jj_|j|jj_yrrr~rur8rnew_embeddingss r0set_output_embeddingsz(BertForPreTraining.set_output_embeddings),'5$$2$7$7!rrrrxrurrlabelsnext_sentence_labelrrPrQrc  | | n|jj} |j||||||| | |  } | dd\} }|j| |\}}d}|u|st }||j d|jj |j d}||j dd|j d}||z}| s||f| ddz}||f|zS|St|||| j| jS)am labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForPreTraining.from_pretrained("google-bert/bert-base-uncased") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` NrrxrurrrrPrQr<rv)rrrrrV) r]rrrr rr~rrrV)rrrrxrurrrrrrPrQrrrkrr total_lossloss_fctmasked_lm_lossnext_sentence_lossrs r0rzBertForPreTraining.forward-sGV&1%L 'tnL d^L  uU\\"$<< =L L rrzP Bert Model with a `language modeling` head on top for CLM fine-tuning. c"eZdZddgZfdZdZdZe ddee jdee jdee jd ee jd ee jd ee jd ee jd ee jdee jdee e jdee dee dee dee de ee jeffdZdZxZS)BertLMHeadModelzcls.predictions.decoder.biasrct|||jstj dt |d|_t||_|jy)NzLIf you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`Fr r{r|rrBwarningrrr|rrrs r0r|zBertLMHeadModel.__init__sL    NNi jf> "6* rcB|jjjSrrrxs r0rz%BertLMHeadModel.get_output_embeddingsrrc||jj_|j|jj_yrrrs r0rz%BertLMHeadModel.set_output_embeddingsrrrrrxrurrrrrrOrrrPrQrc ||n|jj}| d} |j||||||||| | | | | }|d}|j|}d}| )|j|| |jj fi|}|s|f|ddz}||f|zS|St |||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NF) rrxrurrrrrOrrrPrQrr<)rlogitsrOrrVrW) r]rrr loss_functionr~rrOrrVrW)rrrrxrurrrrrrOrrrPrQ loss_kwargsrrrlm_lossrs r0rzBertLMHeadModel.forwards2&1%.s.nU_j--aZ=N=N1OPns58)r[)rrOrreordered_past layer_pasts ` r0_reorder_cachezBertLMHeadModel._reorder_caches=) J ncmnn N r)NNNNNNNNNNNNNN)rrrrr|rrr rrYrrrrrrrrrrs@r0rr}s} 9:Z[ ,8-11515/3,0048<9=)-8<$(,0/3&*> ELL)> !.> !. > u||, > ELL) >   ->  ( 5> !) 6> &> "$u||"45> D>> $D>> 'tn> d^> " uU\\"$EE F#> > @rrceZdZddgZfdZdZdZe ddee jdee jdee jd ee jd ee jd ee jd ee jd ee jdee jdee dee dee de e e jeffdZddZede fdZxZS)BertForMaskedLMrrct|||jrtj dt |d|_t||_|jy)NzkIf you want to use `BertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Frrrs r0r|zBertForMaskedLM.__init__sR     NN1  f> "6* rcB|jjjSrrrxs r0rz%BertForMaskedLM.get_output_embeddingsrrc||jj_|j|jj_yrrrs r0rz%BertForMaskedLM.set_output_embeddingsrrrrrxrurrrrrrrPrQrc | | n|jj} |j||||||||| | |  } | d}|j|}d}| Ft }||j d|jj | j d}| s|f| ddz}||f|zS|St||| j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` N) rrxrurrrrrrPrQrrvr<rrrrV) r]rrrr rr~rrrV)rrrrxrurrrrrrrPrQrrrrrrs r0rzBertForMaskedLM.forwards.&1% &.IIrcy)z Legacy correction: BertForMaskedLM can't call `generate()` from `GenerationMixin`, even though it has a `prepare_inputs_for_generation` method. Fr-)rs r0 can_generatezBertForMaskedLM.can_generateFs r) NNNNNNNNNNNNr)rrrrr|rrr rrYrrrrrrr classmethodrrrs@r0rrsj46VW ,8-11515/3,0048<9=)-,0/3&*7 ELL)7 !.7 !. 7 u||, 7 ELL) 7   -7  ( 57 !) 67 &7 $D>7 'tn7 d^7  uU\\"N2 37 7 rJ TrrzT Bert Model with a `next sentence prediction (classification)` head on top. c^eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eed eed eed e e eje ffd Z xZ S)BertForNextSentencePredictionct||t||_t ||_|j yr)r{r|rrrrrrs r0r|z&BertForNextSentencePrediction.__init__Us4  f% "6* rrrrxrurrrrrPrQrc d| vr+tjdt| jd}| | n|jj } |j |||||||| |  } | d} |j| }d}|2t}||jdd|jd}| s|f| ddz}||f|zS|St||| j| jS) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForNextSentencePrediction.from_pretrained("google-bert/bert-base-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` rzoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nrr#rvr<r) warningswarn FutureWarningpopr]rrrr rrrrV)rrrrxrurrrrrPrQkwargsrrkseq_relationship_scoresrrrs r0rz%BertForNextSentencePrediction.forward^sT !F * MM%  ZZ 56F%0%Q 'tnQ d^Q  uU\\"$?? @Q Q rr z Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c^eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eed eed eed e e eje ffd Z xZ S)BertForSequenceClassificationcnt|||j|_||_t ||_|j |j n |j}tj||_ tj|j|j|_ |jyr)r{r| num_labelsr]rrclassifier_dropoutrr rrrrr;rrr]rrs r0r|z&BertForSequenceClassification.__init__s   ++ f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rrrrxrurrrrrPrQrc @| | n|jj} |j|||||||| |  } | d} |j| } |j | } d}||jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|j dk(r&|| j|j}n|| |}n|jj dk(r=t}|| jd|j |jd}n,|jj dk(rt}|| |}| s| f| ddz}||f|zS|St!|| | j"| j$ S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr# regressionsingle_label_classificationmulti_label_classificationrvr<r)r]rrrr; problem_typerrzrYrrTr squeezer rr rrrV)rrrrxrurrrrrPrQrrkrrrrs r0rz%BertForSequenceClassification.forwards(&1%E 'tnE d^E  uU\\"$<< =E E rrc^eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eed eed eed e e eje ffd Z xZ S)BertForMultipleChoicec*t||t||_|j |jn |j }t j||_t j|jd|_ |jy)Nr#) r{r|rrrrr rrrrr;rrs r0r|zBertForMultipleChoice.__init__su  f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rrrrxrurrrrrPrQrc L| | n|jj} ||jdn|jd} |!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|1|jd|j d|j dnd}|j |||||||| |  } | d} |j | } |j| }|jd| }d}|t}|||}| s|f| ddz}||f|zS|St||| j| jS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr#rvrrr<r) r]rrVrrrrr;r rrrV)rrrrxrurrrrrPrQ num_choicesrrkrreshaped_logitsrrrs r0rzBertForMultipleChoice.forward!sX&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  )) ))%'/!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rr)rrrr|r rrYrrrrrrrrs@r0r!r!s -11515/3,004)-,0/3&*X ELL)X !.X !. X u||, X ELL) X   -X &X $D>X 'tnX d^X  uU\\"$== >X X rr!c^eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eed eed eed e e eje ffd Z xZ S)BertForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyNFr)r{r|rrrrrr rrrrr;rrs r0r|z#BertForTokenClassification.__init__s   ++f> )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rrrrxrurrrrrPrQrc | | n|jj} |j|||||||| |  } | d} |j| } |j | } d}|2 'tn2 d^2  uU\\"$99 :2 2 rr'c~eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)BertForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyr)) r{r|rrrr rr qa_outputsrrs r0r|z!BertForQuestionAnswering.__init__sU   ++f> ))F$6$68I8IJ rrrrxrurrstart_positions end_positionsrrPrQrc (| | n|jj} |j||||||| | |  } | d} |j| }|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| jS) Nrrr#rvr) ignore_indexr<)r start_logits end_logitsrrV)r]rrr.rMrrrSrclampr rrrV)rrrrxrurrr/r0rrPrQrrrr3r4r ignored_indexr start_lossend_lossrs r0rz BertForQuestionAnswering.forwards&1% ELL)> !.> !. > u||, > ELL) >   -> "%,,/>   -> $D>> 'tn> d^>  uU\\"$@@ A> > rr,) rr!r rr,rr'r4rrrrl)SrrrDr  dataclassesrtypingrrrrrYtorch.utils.checkpoint packagingrr torch.nnr r r activationsr generationrmodeling_attn_mask_utilsrrmodeling_outputsrrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrr r!r"configuration_bertr$ get_loggerrrBrlModulernrrrrrr&r0r4rHrermrrr|rrrrrrrrr rr!r'r,__all__r-rr0rIs  !// AA!)w   .llLL*   H %FR=RYY=@C CLb-bJ RYY  ! 0BII0f ryy   S SlZ "))Z z  "))"299.!bii!&bii& 9299 9%/%%4 :{: :B  q #q  q h ` ,`  ` F ])?]  ]@i)iiX \ $7\  \ ~ V $7V  V rg /g g TB !4B B JJ 2J J Z r