UhddlmZmZmZmZddlZddlmZddlmZddl m Z m Z m Z m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-e*r ddl.m/Z/ddl0m1Z1e+jde3Z4edGddejjZ6GddejjZ7dZ8d?dZ9dejtde;dejtfdZ< d@d ejjd!ejtd"ejtd#ejtd$eejtd%e=d&e=fd'Z>Gd(d)ejjZ?Gd*d+eZ@e(Gd,d-e#ZAGd.d/ejjZBe(Gd0d1eAZCGd2d3ee'ZDe(Gd4d5eAeZEe(d67Gd8d9eAZFe(Gd:d;eAZGe(Gd<d=eAZHgd>ZIy)A)CallableOptionalTupleUnionN)nn)ACT2FN)Cache DynamicCacheSlidingWindowCache StaticCache)GenerationMixin)use_kernel_forward_from_hub)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) Qwen3Config) BlockMask)make_flex_block_causal_maskRMSNormc,eZdZdfd ZdZdZxZS) Qwen3RMSNormct|tjt j ||_||_y)z; Qwen3RMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s z/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.pyr+zQwen3RMSNorm.__init__;s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor-float32powmeanrsqrtr0r/)r1 hidden_states input_dtypevariances r5forwardzQwen3RMSNorm.forwardCsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r6c^t|jjd|jS)Nz, eps=)tupler/shaper0r1s r5 extra_reprzQwen3RMSNorm.extra_reprJs* ))*+6$2G2G1HIIr6)gư>)__name__ __module__ __qualname__r+rDrI __classcell__r4s@r5r(r(9s$;Jr6r(c$eZdZfdZdZxZS)Qwen3MLPct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFbias) r*r+configr2intermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnr1rUr4s r5r+zQwen3MLP.__init__Os  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r6c|j|j|j||j|z}|SN)rZr\rXrY)r1xrZs r5rDzQwen3MLP.forwardYs6NN4;;t~~a/@#ADLLQRO#ST r6)rJrKrLr+rDrMrNs@r5rPrPNs 0r6rPc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr9r8dim)rGr-cat)r`x1x2s r5 rotate_halfrg^sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r6c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerg)qkcossin position_ids unsqueeze_dimq_embedk_embeds r5apply_rotary_pos_embrresY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r6rAn_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r"N)rGexpandreshape)rArsbatchnum_key_value_headsslenhead_dims r5 repeat_kvr|so 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr6modulequerykeyvalueattention_maskscalingdropoutc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr8rr9)rcr;)ptrainingr")r|num_key_value_groupsr-matmul transposerGr functionalsoftmaxr=r<r;rr contiguous) r}r~rrrrrkwargs key_states value_states attn_weights causal_mask attn_outputs r5eager_attention_forwardrs3 ; ;$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  #4==f6I6IJ "4==f6I6IJ $33 KK * * %5t<H$++"?"??"&D @r6rAposition_embeddingsrpast_key_valuecache_positionrrtc |jdd}g|d|j}|j|j|j |j dd} |j |j|j |j dd} |j|j |j dd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|jjdk7r^|jjdk(r(|jddrt j#d nt$|jj}||| | | |f|j&sd n |j(|j*|j,d |\}}|j.g|dj1}|j3|}||fS) Nr9r"r8)rmrlreagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrr)rGr{rrviewrrrrrrupdaterrrU_attn_implementationgetlogger warning_oncerrrrrrwrr)r1rArrrrr input_shape hidden_shape query_statesrrrlrm cache_kwargsattention_interfacerrs r5rDzQwen3Attention.forwards$))#2. 88b8$--8 {{4;;}#=#B#B<#PQ[[\]_`a [[]!;!@!@!NOYYZ[]^_ {{=166|DNNqRST &S#7 jRUWZ#[ j  %#&snUL'5'<'d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL.. %  % ! \*k));;;;FFH kk+. L((r6)NN)rJrKrL__doc__r#intr+r-Tensorrrr LongTensorrrrDrMrNs@r5rrsG'{'s'J+/59 0)||0)#5<<#=>0)!. 0) ! 0) !!1!12 0)-.0) u||Xell3XeELL>Q5RR S0)r6rcpeZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee ejejfd e ed e ejee ejejfffdZxZS)Qwen3DecoderLayerrUrct||j|_t|||_t ||_t|j|j|_ t|j|j|_ |jr4|jdk7r$tjd|jdyyy)N)rUrrflash_attention_2z=Sliding Window Attention is enabled but not implemented for `z)`; unexpected results may be encountered.)r*r+r2r self_attnrPmlpr(rinput_layernormpost_attention_layernormrrrrrs r5r+zQwen3DecoderLayer.__init__s !--'vKF#+F,>,>FDWDWX(4V5G5GVM`M`(a%  ! !f&A&AEX&X   OPVPkPkOlm99 'Y !r6rArrnrr use_cacherrrrtc |} |j|}|jd||||||||d| \}} | |z}|} |j|}|j|}| |z}|f} |r| | fz } | S)N)rArrnrrrrr)rrrr) r1rArrnrrrrrrresidualself_attn_weightsoutputss r5rDzQwen3DecoderLayer.forward s!,,]; ,:4>> , ')%)/) 3 ,  , ( (!=0 !55mD /  =0  "  )+ +Gr6)NNNFFNN)rJrKrLr#rr+r-rrrr boolrrr FloatTensorrDrMrNs@r5rrs  { s $2637*.,1$)59KO'||'!.'u//0 ' ! ' $D> 'D>'!!1!12'&eELL%,,,F&GH'-.' u  (51B1BEDUDU1U+V"WW X'r6rcFeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZdZdZy)Qwen3PreTrainedModelmodelTrpast_key_valuesc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyt|tr&|j j jdyy)Nr)r?stdg?)rUinitializer_range isinstancerrWr/datanormal_rTzero_ Embedding padding_idxr(fill_)r1r}rs r5 _init_weightsz"Qwen3PreTrainedModel._init_weightsEskk++ fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  - MM   $ $S ).r6N)rJrKrLr# config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_attention_backendrrr6r5rr6sSL&*#,-#4"5!N  $!"& *r6rc^eZdZddeffd Zej edZxZ S)Qwen3RotaryEmbeddingrUct|t|drG|j;|jj d|jj d|_nd|_|j |_|j |_||_ t|j |_ |j|j|\}|_ |jd|d|j|_y)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r*r+hasattrrrrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrUr rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r1rUdevicerr4s r5r+zQwen3RotaryEmbedding.__init__Ts  6> *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.r_) rJrKrLr#r+r-no_gradrrDrMrNs@r5rrSs3/{/"U]]_ < ?ab b   --i8M 0*nO  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L.. M>?L] & #oom\J#7BD0d![[)H4;;+H+HI 6M#!m%55!) *)."3#-$7 $ M*!,M =#3"55' 6* -0  -!1 1 &+/8Od+%   r6r$ input_tensorc l|jjdk(rS|H|F|dddfjj|j dk7}|r t d|d|vr|Sy|jjdk(r't |tjr t|}|S||jnd}t |t}t |t} |jjdk(r?|s=| s;|s9tj||||jj|j ry|j"} tj$| j&} |j(d } | s|r|j+} n1t |tjr|j(dn|| zd z} |j-|| | | ||j(d|j| }|jjdk(r2|0|j.j0d vr|stj2|| }|S) Nrr9rzYou are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Qwen3. Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input. rflex_attentionr)r past_key_values_lengthr is_trainingr")sequence_length target_lengthr;r batch_sizerUr)cudaxpunpu)rUrsumitemsizer'rr-rr%r(r r r_ignore_causal_mask_sdparrr;finfominrGget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrr_unmask_unattended)r1rr0rrris_padding_rightr+using_static_cacheusing_sliding_window_cacher; min_dtyper5r6rs r5r*zQwen3Model._update_causal_masks: ;; + +/B B)o.I#1!R%#8#<#<#>#C#C#EIZIZI\]^I_#_ #$a )c^.C%% ;; + +/? ?.%,,7!<^!L! ! @O?Z?99;`a'E%/AS%T" KK , , 6'+E%%>>*'7#{{99 MM ""KK&** &,,Q/ %);+??AM nell;$$R(%7!; PP +')#))!,;;+Q   KK , , 6*%%**.DD% 1CCKQZ[Kr6r5r6r;r7c||jdk(r|}|Stj|j} tj||f| ||j }tj ||j |jddkD} |j} t| ddrs| jgt|tr||kDrRtj ||j |jdd| jz k} | j| || z}|ddddddfj|ddd}||j}|j d|kDr |ddd|f}|j d} |ddddddd| f|ddddddfj#|j z}|d k(}|ddddddd| fj%|| |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. config (`Qwen3Config`): The model's configuration class past_key_values (`Cache`): The cache class that is being used currently to generate N) fill_valuer;rr$r9r"rTr)rcr-r?r@fullrr)rwget_text_configrrrr bitwise_or_rvclonerGr< masked_fill)rr5r6r;rr7rUrrrGdiagonal_attend_mask text_configsliding_attend_mask mask_length padding_masks r5rBz@Qwen3Model._prepare_4d_causal_attention_mask_with_cache_positionCs%B  %.*<*<*>!*C(K@= E*..I** -0Ye\j\q\qK$)<< nF[F[#\_m_u_uA`$ !002K{$8$?KD^D^Dj"/3EF/\iJi*/,,}^MbMb*c&..r15 8R8RR+')445HI / /K%dD!Q&67>>z1bRTUK))//1 !''+m;%3A~ ~4E%FN,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r6 NNNNNNNNN)F)rJrKrLr#r+rrrrrr-rrr rrrrrrDrr* staticmethodrr;rBrMrNs@r5r r us{ !"151537+/59$(,0/359\ E,,-\ !.\ u//0 \ "% \   1 12 \ D>\ $D>\ 'tn\ !!1!12\ $$89\  !\ \ H#( TellK78TllT T  T  TlB BBB{{ B  B  BBBBr6r c eZdZy)KwargsForCausalLMN)rJrKrLrr6r5rXrXsr6rXceZdZdgZddiZddgdgfiZfdZdZdZd Z d Z d Z d Z e e dd eej"deej$deej"deedeej(deej"deedeedeedeej"deeej$fdeedefdZxZS)Qwen3ForCausalLMzlm_head.weightlm_head colwise_reprAlogitsct||t||_|j|_t j |j|jd|_|jyrR) r*r+r rrrrWr2r[rr]s r5r+zQwen3ForCausalLM.__init__sU  '  ++yy!3!3V5F5FUS  r6c.|jjSr_rrrHs r5rz%Qwen3ForCausalLM.get_input_embeddingszz&&&r6c&||j_yr_r`rs r5rz%Qwen3ForCausalLM.set_input_embeddings"' r6c|jSr_r[rHs r5get_output_embeddingsz&Qwen3ForCausalLM.get_output_embeddingss ||r6c||_yr_re)r1new_embeddingss r5set_output_embeddingsz&Qwen3ForCausalLM.set_output_embeddingss % r6c||_yr_r)r1decoders r5 set_decoderzQwen3ForCausalLM.set_decoders  r6c|jSr_rkrHs r5 get_decoderzQwen3ForCausalLM.get_decoders zzr6rrrnrr labelsrrr!rlogits_to_keeprrtc ||n|jj}| | n|jj} |jd|||||||| | d | } | j}t | t r t| dn| }|j|dd|ddf}d}|*|jd|||jjd| }t||| j| j| jS)a^ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Qwen3ForCausalLM >>> model = Qwen3ForCausalLM.from_pretrained("Qwen/Qwen3-8B") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrnrr rrr!r)r]rprlossr]rrAr&r)rUrr!rr%rrslicer[ loss_functionrrrrAr&)r1rrrnrr rprrr!rrqrrrA slice_indicesr]rts r5rDzQwen3ForCausalLM.forwardsN2C1N-TXT_T_TqTq$8$D $++JjJj  ,64:: , )%+'/!5) ,  ,  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD%#33!//))   r6) NNNNNNNNNNr)rJrKrL_tied_weights_keys_tp_plan_pp_planr+rrrfrirmrorrrr-rrr rrrrrrXrrDrMrNs@r5rZrZs*+=)H_-z:;H'(&151537+/59-1$(,0/35934G E,,-G !.G u//0 G "% G   1 12 G ))*G D>G $D>G 'tnG !!1!12G c5<</0G *+G  G G r6rZa The Qwen3 Model transformer with a sequence classification head on top (linear layer). [`Qwen3ForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introceZdZfdZdZdZee ddee jdee jdee jdee dee jd ee jd eed eed eed efdZxZS)Qwen3ForSequenceClassificationct|||j|_t||_t j |j|jd|_|jyrR) r*r+ num_labelsr rrrWr2scorerr]s r5r+z'Qwen3ForSequenceClassification.__init__sS   ++' YYv114??O  r6c.|jjSr_r`rHs r5rz3Qwen3ForSequenceClassification.get_input_embeddingsrar6c&||j_yr_r`rs r5rz3Qwen3ForSequenceClassification.set_input_embeddingsrcr6rrrnrr rprrr!rtc |j|||||||| } | j} |j| } ||jd} n|jd} |jj | dk7r t d|jj d}n|||jj k7j| jtj}tj|jd| jtj}||zjd}n.d}tj|jj d| tj| | j |f}d}||j#| |||j }t%||| j&| j(| j* S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rrnrr rrr!Nrr"z=Cannot handle batch sizes > 1 if no padding token is defined.r9)rr;z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r$)r]rp pooled_logitsrUrs)rr%rrGrUrr'r<rr-int32r)argmaxrrr4rJrvrrrAr&)r1rrrnrr rprrr!transformer_outputsrAr]r7last_non_pad_token non_pad_mask token_indicesrrts r5rDz&Qwen3ForSequenceClassification.forwards*8!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   %%VFR_hlhshs%tD/ /??-;;*55   r6rU)rJrKrLr+rrrrrr-rrr rrrrDrMrNs@r5r}r}s'(151537+/59-1$(,0/3A E,,-A !.A u//0 A "% A   1 12 A ))*A D>A $D>A 'tnA  *A A r6r}ceZdZfdZdZdZee ddee jdee jdee jdee dee jd ee jd eed eed eed efdZxZS)Qwen3ForTokenClassificationct|||j|_t||_t |dd |j }nt |dd |j}nd}tj||_ tj|j|j|_ |jy)Nclassifier_dropouthidden_dropoutg?)r*r+rr rrrrrDropoutrrWr2rr)r1rUrr4s r5r+z$Qwen3ForTokenClassification.__init___s   ++' 6/ 6 B!'!:!:  V-t 4 @!'!6!6 !$ zz"45 YYv1163D3DE  r6c.|jjSr_r`rHs r5rz0Qwen3ForTokenClassification.get_input_embeddingsorar6c&||j_yr_r`rs r5rz0Qwen3ForTokenClassification.set_input_embeddingsrrcr6rrrnrr rprrr!rtc |j|||||||| } | j} |j| } |j| } d} ||j | ||j } t | | | j| jS)rrN)rtr]rAr&) rr%rrrvrUrrAr&)r1rrrnrr rprrr!rsequence_outputr]rts r5rDz#Qwen3ForTokenClassification.forwardus*,0:: )%+'/!5,6 , "33,,7O,  %%ffdkkBD$!//))   r6rU)rJrKrLr+rrrrrr-rrr rrrrDrMrNs@r5rr]s '(151537+/59-1$(,0/3* E,,-* !.* u//0 * "% *   1 12 * ))** D>* $D>* 'tn*  * * r6rc4eZdZdZfdZdZdZee dde e jde e jde e jde e d e e jd e e jd e e jd e ed e edefdZxZS)Qwen3ForQuestionAnswering transformerct||t||_t j |j d|_|jy)Nr8) r*r+r rrrWr2 qa_outputsrr]s r5r+z"Qwen3ForQuestionAnswering.__init__sA  %f-))F$6$6: r6c.|jjSr_rrrHs r5rz.Qwen3ForQuestionAnswering.get_input_embeddingss,,,r6c&||j_yr_rrs r5rz.Qwen3ForQuestionAnswering.set_input_embeddingss(-%r6rrrnrr start_positions end_positionsrr!rtc |j||||||| } | j} |j| } | jdd\}}|j dj }|j dj }d}|||j ||||fi| }t|||| j| jS)N)rrnrr rr!r"r9rb)rt start_logits end_logitsrAr&) rr%rsplitsqueezerrvrrAr&)r1rrrnrr rrrr!rrrr]rrrts r5rDz!Qwen3ForQuestionAnswering.forwards,0+;+; )%+'/!5,<, "331#)<<r<#: j#++B/::< ''+668   &=+D%4%%lJQ^ibhiD+%!!//))   r6rU)rJrKrLrr+rrrrrr-rrr rrrrDrMrNs@r5rrs%-.151537+/596:48,0/3( E,,-( !.( u//0 ( "% (   1 12 ( "%"2"23(   0 01( $D>( 'tn(  &( ( r6r)rZrr rr}r)Nr")r)Jtypingrrrrr-r activationsr cache_utilsr r r r generationr integrationsrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr r!configuration_qwen3r#!torch.nn.attention.flex_attentionr$integrations.flex_attentionr% get_loggerrJrModuler(rPrgrrrrr|rrrrrrr rXrZr}rr__all__rr6r5rsB,43 !OO)7>B9LF&hh, !;J   H %Y'J299J(J( ryy  (6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%4S)RYYS)l727t*?**8<299i +_i i X  S %9S  S lC "6C C L; 4; ; | r6