Uh¦ddlmZmZmZmZddlZddlmZddlmZddl m Z m Z m Z m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-e*r ddl.m/Z/ddl0m1Z1e+jde3Z4GddejjZ6dZ7d?dZ8dejrde:dejrfdZ; d@dejjdejrdejrd ejrd!eejrd"eed'Gd(d)ejjZ?Gd*d+eZ@e(Gd,d-e#ZAGd.d/ejjZBe(Gd0d1eAZCGd2d3ee'ZDe(Gd4d5eAeZEe(d67Gd8d9eAZFe(Gd:d;eAZGe(Gd<d=eAZHgd>ZIy)A)CallableOptionalTupleUnionN)nn)ACT2FN)Cache DynamicCacheSlidingWindowCache StaticCache)GenerationMixin)use_kernel_forward_from_hub)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) Qwen2Config) BlockMask)make_flex_block_causal_maskc$eZdZfdZdZxZS)Qwen2MLPct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFbias) super__init__config hidden_sizeintermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnselfr. __class__s z/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/qwen2/modeling_qwen2.pyr-zQwen2MLP.__init__+s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ c|j|j|j||j|z}|SN)r4r6r2r3)r8xr4s r:forwardzQwen2MLP.forward5s6NN4;;t~~a/@#ADLLQRO#ST r;)__name__ __module__ __qualname__r-r? __classcell__r9s@r:r'r'*s 0r;r'c|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)r>x1x2s r: rotate_halfrO:sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r;c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerO)qkcossin position_ids unsqueeze_dimq_embedk_embeds r:apply_rotary_pos_embrZAsY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r; hidden_statesn_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r"N)rJexpandreshape)r[r\batchnum_key_value_headsslenhead_dims r: repeat_kvre\so 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr;modulequerykeyvalueattention_maskscalingdropoutc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)NrGrrF)rIdtype)ptrainingr")renum_key_value_groupsrKmatmul transposerJr functionalsoftmaxfloat32tororlrq contiguous) rfrgrhrirjrkrlkwargs key_states value_states attn_weights causal_mask attn_outputs r:eager_attention_forwardrhs3 ; ;$>&B\B\$\!}}d* !'!9!9ii 2 2F4N4NQUQ^Q^4^eij ii 2 2F4N4NQUQ^Q^4^eij ii 2 2F4N4NQUQ^Q^4^eij ii : :T]] JFL^L^ejk r;r[position_embeddingsrjpast_key_valuecache_positionrzr]c |jdd}g|d|j}|j|j|j dd} |j |j|j dd} |j |j|j dd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } d}|jjrPt|jdd9|j|jjk\r|jj}t}|jj dk7r^|jj dk(r(|j#dd rt$j'd nt(|jj }||| | | |f|j*sd n |j,|j.|d |\}}|j0g|dj3}|j5|}||fS) NrFr"rG)rUrTrsliding_windoweagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rlrkr)rJrdrviewrtrrrZupdaterr.use_sliding_windowrmax_window_layersrr_attn_implementationgetlogger warning_oncerrqrrkr`ryr)r8r[rrjrrrz input_shape hidden_shape query_statesr{r|rTrU cache_kwargsrattention_interfacerr}s r:r?zQwen2Attention.forwards/$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &S#7 jRUWZ#[ j  %#&snUL'5'<'d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL) %  % ! \*k));;;;FFH kk+. L((r;)NN)r@rArB__doc__r#intr-rKTensorrrr LongTensorrrr?rCrDs@r:rrsG l{ ls l&+/59 8)||8)#5<<#=>8)!. 8) ! 8) !!1!12 8)-.8) u||Xell3XeELL>Q5RR S8)r;rRMSNormc,eZdZdfd ZdZdZxZS) Qwen2RMSNormct|tjt j ||_||_y)z; Qwen2RMSNorm is equivalent to T5LayerNorm N)r,r-r ParameterrKonesweightvariance_epsilon)r8r/epsr9s r:r-zQwen2RMSNorm.__init__s1 ll5::k#:; #r;c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrGrFT)keepdim) rorxrKrwpowmeanrsqrtrr)r8r[ input_dtypevariances r:r?zQwen2RMSNorm.forwardsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r;c^t|jjd|jS)Nz, eps=)tuplerrJrr8s r: extra_reprzQwen2RMSNorm.extra_reprs* ))*+6$2G2G1HIIr;)gư>)r@rArBr-r?rrCrDs@r:rrs$;Jr;rcpeZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee ejejfd e ed e ejee ejejfffdZxZS)Qwen2DecoderLayerr.rct||j|_t|||_t ||_t|j|j|_ t|j|j|_ |jr4|jdk7r$tjd|jdyyy)N)r.rrflash_attention_2z=Sliding Window Attention is enabled but not implemented for `z)`; unexpected results may be encountered.)r,r-r/r self_attnr'mlpr rms_norm_epsinput_layernormpost_attention_layernormrrrrrs r:r-zQwen2DecoderLayer.__init__s !--'vKF#+F,>,>FDWDWX(4V5G5GVM`M`(a%  $ $)D)DH[)[   OPVPkPkOlm99 *\ $r;r[rjrVrr use_cacherrrzr]c |} |j|}|jd||||||||d| \}} | |z}|} |j|}|j|}| |z}|f} |r| | fz } | S)N)r[rjrVrrrrr)rrrr) r8r[rjrVrrrrrrzresidualself_attn_weightsoutputss r:r?zQwen2DecoderLayer.forwards!,,]; ,:4>> , ')%)/) 3 ,  , ( (!=0 !55mD /  =0  "  )+ +Gr;)NNNFFNN)r@rArBr#rr-rKrrrr boolrrr FloatTensorr?rCrDs@r:rrs  { s  2637*.,1$)59KO'||'!.'u//0 ' ! ' $D> 'D>'!!1!12'&eELL%,,,F&GH'-.' u  (51B1BEDUDU1U+V"WW X'r;rcFeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZdZdZy)Qwen2PreTrainedModelmodelTrpast_key_valuesc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyt|tr&|j j jdyy)Nr)rstdg?)r.initializer_range isinstancerr1rdatanormal_r+zero_ Embedding padding_idxrfill_)r8rfrs r: _init_weightsz"Qwen2PreTrainedModel._init_weights*skk++ fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  - MM   $ $S ).r;N)r@rArBr# config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_attention_backendrrr;r:rrsSL&*#,-#4"5!N  $!"& *r;rc^eZdZddeffd Zej edZxZ S)Qwen2RotaryEmbeddingr.ct|t|drG|j;|jj d|jj d|_nd|_|j |_|j |_||_ t|j |_ |j|j|\}|_ |jd|d|j|_y)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r,r-hasattrrrrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr.r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r8r.devicerr9s r:r-zQwen2RotaryEmbedding.__init__9s  6> *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+rVinv_freq_expandedposition_ids_expandedrfreqsembrTrUs r:r?zQwen2RotaryEmbedding.forwardJsV!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.r=) r@rArBr#r-rKno_gradrr?rCrDs@r:rr8s3/{/"U]]_ < ?ab b   --i8M 0*nO  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L.. M>?L] & #oom\J#7BD0d![[)H4;;+H+HI 6M#!m%55!) *)."3#-$7 $ M*!,M =#3"55' 6* -0  -!1 1 &+/8Od+%   r;r$ input_tensorc l|jjdk(rS|H|F|dddfjj|j dk7}|r t d|d|vr|Sy|jjdk(r't |tjr t|}|S||jnd}t |t}t |t} |jjdk(r?|s=| s;|s9tj||||jj|j ry|j"} tj$| j&} |j(d } | s|r|j+} n1t |tjr|j(dn|| zd z} |j-|| | | ||j(d|j| }|jjdk(r2|0|j.j0d vr|stj2|| }|S) NrrFrzYou are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Qwen2. Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input. rflex_attentionr)rpast_key_values_lengthr is_trainingr")sequence_length target_lengthror batch_sizer.r)cudaxpunpu)r.rsumitemsizer$rrKrr%r%r r r_ignore_causal_mask_sdparrqrofinfominrJget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrr_unmask_unattended)r8rjr-rrris_padding_rightr(using_static_cacheusing_sliding_window_cachero min_dtyper2r3r~s r:r'zQwen2Model._update_causal_masks: ;; + +/B B)o.I#1!R%#8#<#<#>#C#C#EIZIZI\]^I_#_ #$a )c^.C%% ;; + +/? ?.%,,7!<^!L! ! @O?Z?99;`a'E%/AS%T" KK , , 6'+E%%>>*'7#{{99 MM ""KK&** &,,Q/ %);+??AM nell;$$R(%7!; PP +')#))!,;;+Q   KK , , 6*%%**.DD% 1CCKQZ[Kr;r2r3ror4c||jdk(r|}|Stj|j} tj||f| ||j }tj ||j |jddkD} |j} t| ddrs| jgt|tr||kDrRtj ||j |jdd| jz k} | j| || z}|ddddddfj|ddd}||j}|j d|kDr |ddd|f}|j d} |ddddddd| f|ddddddfj#|j z}|d k(}|ddddddd| fj%|| |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. config (`Qwen2Config`): The model's configuration class past_key_values (`Cache`): The cache class that is being used currently to generate N) fill_valuerorr!rFr"rTr)rIrKr<r=fullrr&r`get_text_configrrrr bitwise_or_r_clonerJrx masked_fill)rjr2r3rorr4r.rr~rDdiagonal_attend_mask text_configsliding_attend_mask mask_length padding_masks r:r?z@Qwen2Model._prepare_4d_causal_attention_mask_with_cache_position(s%B  %.*<*<*>!*C(K@= E*..I** -0Ye\j\q\qK$)<< nF[F[#\_m_u_uA`$ !002K{$8$?KD^D^Dj"/3EF/\iJi*/,,}^MbMb*c&..r15 8R8RR+')445HI / /K%dD!Q&67>>z1bRTUK))//1 !''+m;%3A~ ~4E%FN,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r; NNNNNNNNN)F)r@rArBr#r-rrrrrrKrrr rrrrrr?rr' staticmethodrror?rCrDs@r:r r Zs{ !"151537+/59$(,0/359\ E,,-\ !.\ u//0 \ "% \   1 12 \ D>\ $D>\ 'tn\ !!1!12\ $$89\  !\ \ H#( TellK78TllT T  T  TlB BBB{{ B  B  BBBBr;r c eZdZy)KwargsForCausalLMN)r@rArBrr;r:rUrUnsr;rUceZdZdgZddiZddgdgfiZfdZdZdZd Z d Z d Z d Z e e dd eej"deej$deej"deedeej(deej"deedeedeedeej"deeej$fdeedefdZxZS)Qwen2ForCausalLMzlm_head.weightlm_head colwise_repr[logitsct||t||_|j|_t j |j|jd|_|jyr)) r,r-r rr rr1r/rXrr7s r:r-zQwen2ForCausalLM.__init__wsU  '  ++yy!3!3V5F5FUS  r;c.|jjSr=rr rs r:rz%Qwen2ForCausalLM.get_input_embeddingszz&&&r;c&||j_yr=r]rs r:rz%Qwen2ForCausalLM.set_input_embeddings"' r;c|jSr=rXrs r:get_output_embeddingsz&Qwen2ForCausalLM.get_output_embeddingss ||r;c||_yr=rb)r8new_embeddingss r:set_output_embeddingsz&Qwen2ForCausalLM.set_output_embeddingss % r;c||_yr=r)r8decoders r: set_decoderzQwen2ForCausalLM.set_decoders  r;c|jSr=rhrs r: get_decoderzQwen2ForCausalLM.get_decoders zzr;rrjrVrrlabelsrrrrlogits_to_keeprzr]c ||n|jj}| | n|jj} |jd|||||||| | d | } | j}t | t r t| dn| }|j|dd|ddf}d}|*|jd|||jjd| }t||| j| j| jS)at labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Qwen2ForCausalLM >>> model = Qwen2ForCausalLM.from_pretrained("meta-qwen2/Qwen2-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-qwen2/Qwen2-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrjrVrrrrrr)rZrmr lossrZrr[r#r)r.rrrr"rrslicerX loss_functionr rrr[r#)r8rrjrVrrrmrrrrrnrzrr[ slice_indicesrZrqs r:r?zQwen2ForCausalLM.forwardsN2C1N-TXT_T_TqTq$8$D $++JjJj  ,64:: , )%+'/!5) ,  ,  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD%#33!//))   r;) NNNNNNNNNNr)r@rArB_tied_weights_keys_tp_plan_pp_planr-rrrcrfrjrlrrrrKrrr rrrrrrUrr?rCrDs@r:rWrWqs*+=)H_-z:;H'(&151537+/59-1$(,0/35934G E,,-G !.G u//0 G "% G   1 12 G ))*G D>G $D>G 'tnG !!1!12G c5<</0G *+G  G G r;rWa The Qwen2 Model transformer with a sequence classification head on top (linear layer). [`Qwen2ForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introceZdZfdZdZdZee ddee jdee jdee jdee dee jd ee jd eed eed eed efdZxZS)Qwen2ForSequenceClassificationct|||j|_t||_t j |j|jd|_|jyr)) r,r- num_labelsr rrr1r/scorerr7s r:r-z'Qwen2ForSequenceClassification.__init__sS   ++' YYv114??O  r;c.|jjSr=r]rs r:rz3Qwen2ForSequenceClassification.get_input_embeddingsr^r;c&||j_yr=r]rs r:rz3Qwen2ForSequenceClassification.set_input_embeddingsr`r;rrjrVrrrmrrrr]c |j|||||||| } | j} |j| } ||jd} n|jd} |jj | dk7r t d|jj d}n|||jj k7j| jtj}tj|jd| jtj}||zjd}n.d}tj|jj d| tj| | j |f}d}||j#| |||j }t%||| j&| j(| j* S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rjrVrrrrrNrr"z=Cannot handle batch sizes > 1 if no padding token is defined.rF)rroz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r!)rZrm pooled_logitsr.rp)rr"r}rJr.r r$rxrrKint32r&argmaxrrr9r@rsrrr[r#)r8rrjrVrrrmrrrtransformer_outputsr[rZr4last_non_pad_token non_pad_mask token_indicesrrqs r:r?z&Qwen2ForSequenceClassification.forwards*8!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   %%VFR_hlhshs%tD/ /??-;;*55   r;rR)r@rArBr-rrrrrrKrrr rrrr?rCrDs@r:rzrzs'(151537+/59-1$(,0/3A E,,-A !.A u//0 A "% A   1 12 A ))*A D>A $D>A 'tnA  *A A r;rzceZdZfdZdZdZee ddee jdee jdee jdee dee jd ee jd eed eed eed efdZxZS)Qwen2ForTokenClassificationct|||j|_t||_t |dd |j }nt |dd |j}nd}tj||_ tj|j|j|_ |jy)Nclassifier_dropouthidden_dropoutg?)r,r-r|r rrrrrDropoutrlr1r/r}r)r8r.rr9s r:r-z$Qwen2ForTokenClassification.__init__Ds   ++' 6/ 6 B!'!:!:  V-t 4 @!'!6!6 !$ zz"45 YYv1163D3DE  r;c.|jjSr=r]rs r:rz0Qwen2ForTokenClassification.get_input_embeddingsTr^r;c&||j_yr=r]rs r:rz0Qwen2ForTokenClassification.set_input_embeddingsWr`r;rrjrVrrrmrrrr]c |j|||||||| } | j} |j| } |j| } d} ||j | ||j } t | | | j| jS)rrN)rqrZr[r#) rr"rlr}rsr.rr[r#)r8rrjrVrrrmrrrrsequence_outputrZrqs r:r?z#Qwen2ForTokenClassification.forwardZs*,0:: )%+'/!5,6 , "33,,7O,  %%ffdkkBD$!//))   r;rR)r@rArBr-rrrrrrKrrr rrrr?rCrDs@r:rrBs '(151537+/59-1$(,0/3* E,,-* !.* u//0 * "% *   1 12 * ))** D>* $D>* 'tn*  * * r;rc4eZdZdZfdZdZdZee dde e jde e jde e jde e d e e jd e e jd e e jd e ed e edefdZxZS)Qwen2ForQuestionAnswering transformerct||t||_t j |j d|_|jy)NrG) r,r-r rrr1r/ qa_outputsrr7s r:r-z"Qwen2ForQuestionAnswering.__init__sA  %f-))F$6$6: r;c.|jjSr=rr rs r:rz.Qwen2ForQuestionAnswering.get_input_embeddingss,,,r;c&||j_yr=rrs r:rz.Qwen2ForQuestionAnswering.set_input_embeddingss(-%r;rrjrVrrstart_positions end_positionsrrr]c |j||||||| } | j} |j| } | jdd\}}|j dj }|j dj }d}|||j ||||fi| }t|||| j| jS)N)rjrVrrrrr"rFrH)rq start_logits end_logitsr[r#) rr"rsplitsqueezeryrsrr[r#)r8rrjrVrrrrrrrzrrrZrrrqs r:r?z!Qwen2ForQuestionAnswering.forwards,0+;+; )%+'/!5,<, "331#)<<r<#: j#++B/::< ''+668   &=+D%4%%lJQ^ibhiD+%!!//))   r;rR)r@rArBrr-rrrrrrKrrr rrrr?rCrDs@r:rrs%-.151537+/596:48,0/3( E,,-( !.( u//0 ( "% (   1 12 ( "%"2"23(   0 01( $D>( 'tn(  &( ( r;r)rr rWrzrr)Nr")r)JtypingrrrrrKr activationsr cache_utilsr r r r generationr integrationsrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr r!configuration_qwen2r#!torch.nn.attention.flex_attentionr$integrations.flex_attentionr% get_loggerr@rModuler'rOrZrrrerrrrrrrr rUrWrzrr__all__rr;r:rsB43 !OO)7>B9LF&hh, !;J   H % ryy  (6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%4I)RYYI)XY'J299J(J(525p*?**8<299i +_i i X  S %9S  S lC "6C C L; 4; ; | r;