UhdZddlmZmZmZddlZddlZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZmZd d lmZd d lmZmZmZmZmZmZmZmZmZd dl m!Z!ddl"m#Z#ejHe%Z&dZ'GddeZ(GddeZ)GddeZ*GddeZ+Gdde!Z,Gdde eZ-GddeZ.Gd d!eZ/Gd"d#eZ0Gd$d%eZ1gd&Z2y)'zPyTorch Qwen3 model.)CallableOptionalTupleN)Cache)FlashAttentionKwargs)CausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONS)Unpack) LossKwargslogging)GemmaMLP) LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaForQuestionAnsweringLlamaForSequenceClassificationLlamaForTokenClassification LlamaRMSNormapply_rotary_pos_embeager_attention_forward) MistralModel) Qwen3Configz Qwen/Qwen3-8Bc eZdZy) Qwen3RMSNormN__name__ __module__ __qualname__y/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/qwen3/modular_qwen3.pyrr1r#rc eZdZy)Qwen3MLPNrr"r#r$r'r'5r%r#r'c2eZdZdedeffd Z d dejdeejejfde ejde e de ejd e e d eeje eje eejffd ZxZS) Qwen3Attentionconfig layer_idxct|||t|j|j|_t|j|j|_|j|_|jjr:t|jdd#|j|jjk\sd|_yy)N)epssliding_window) super__init__rhead_dim rms_norm_epsq_normk_normr.r*use_sliding_windowgetattrr+max_window_layersselfr*r+ __class__s r$r0zQwen3Attention.__init__:s +"4==f6I6IJ "4==f6I6IJ $33 KK * * %5t<H$++"?"??"&D @r# hidden_statesposition_embeddingsattention_maskpast_key_valuecache_positionkwargsreturnc |jdd}g|d|j}|j|j|j |j dd} |j |j|j |j dd} |j|j |j dd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|jjdk7r^|jjdk(r(|jddrt j#d nt$|jj}||| | | |f|j&sd n |j(|j*|j,d |\}}|j.g|dj1}|j3|}||fS) Nrr)sincosr?eagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.g)dropoutscalingr.)shaper1r3q_projview transposer4k_projv_projrupdater+rr*_attn_implementationgetlogger warning_oncer trainingattention_dropoutrJr.reshape contiguouso_proj)r9r;r<r=r>r?r@ input_shape hidden_shape query_states key_states value_statesrErD cache_kwargsattention_interface attn_output attn_weightss r$forwardzQwen3Attention.forwardFs$))#2. 88b8$--8 {{4;;}#=#B#B<#PQ[[\]_`a [[]!;!@!@!NOYYZ[]^_ {{=166|DNNqRST &S#7 jRUWZ#[ j  %#&snUL'5'<'d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL.. %  % ! \*k));;;;FFH kk+. L((r#)NN)rr r!rintr0torchTensorrrr LongTensorr rrd __classcell__r:s@r$r)r)9s '{ 's '"+/59 0)||0)#5<<#=>0)!. 0) ! 0) !!1!12 0)-.0) u||Xell3XeELL>Q5RR S0)r#r)c(eZdZdedeffd ZxZS)Qwen3DecoderLayerr*r+ct|t|||_t ||_|j r4|jdk7r$tjd|jdyyy)N)r*r+flash_attention_2z=Sliding Window Attention is enabled but not implemented for `z)`; unexpected results may be encountered.) r/r0r) self_attnr'mlpr.rRrTrUr8s r$r0zQwen3DecoderLayer.__init__zsp 'vKF#  ! !f&A&AEX&X   OPVPkPkOlm99 'Y !r#)rr r!rrer0rirjs@r$rlrlys { s  r#rlc eZdZy) Qwen3ModelNrr"r#r$rrrrr%r#rrc eZdZy)KwargsForCausalLMNrr"r#r$rtrtsr#rtc.eZdZdeedeffd ZxZS)Qwen3ForCausalLM super_kwargsrAc "t|di|S)a^ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Qwen3ForCausalLM >>> model = Qwen3ForCausalLM.from_pretrained("Qwen/Qwen3-8B") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```r")r/rd)r9rwr:s r$rdzQwen3ForCausalLM.forwards4w...r#)rr r!r rtr rdrirjs@r$rvrvs%/01/ //r#rvc eZdZy)Qwen3ForSequenceClassificationNrr"r#r$rzrzr%r#rzc eZdZy)Qwen3ForTokenClassificationNrr"r#r$r|r|r%r#r|c eZdZy)Qwen3ForQuestionAnsweringNrr"r#r$r~r~r%r#r~)rvr~rrQwen3PreTrainedModelrzr|)3__doc__typingrrrrftorch.utils.checkpoint cache_utilsrmodeling_flash_attention_utilsrmodeling_outputsr modeling_utilsr processing_utilsr utilsr r gemma.modeling_gemmarllama.modeling_llamarrrrrrrrrmistral.modeling_mistralrconfiguration_qwen3r get_loggerrrT_CHECKPOINT_FOR_DOCrr'r)rlrrrtrvrzr|r~__all__r"r#r$rs,,  B65&(+   4,   H %% <  x =)^=)@ )   ?,j>/'/< %C  "=  9  r#