UhDXdZddlmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZd d lmZd d lmZmZmZmZmZmZmZddlmZej<eZ GddeZ!GddeZ"GddeZ#GddeZ$GddeZ%GddeZ&gdZ'y)zPyTorch BitNet model.)CallableOptionalTupleN)Cache)FlashAttentionKwargs)CausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONS)Unpack)logging)GemmaMLP)LlamaAttentionLlamaDecoderLayerLlamaForCausalLM LlamaModel LlamaRMSNormapply_rotary_pos_embeager_attention_forward) BitNetConfigc eZdZy) BitNetRMSNormN__name__ __module__ __qualname__{/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/bitnet/modular_bitnet.pyrr*rrc*eZdZdeffd ZdZxZS) BitNetMLPconfigcpt||t|j|j|_yN)eps)super__init__rintermediate_size rms_norm_eps ffn_sub_norm)selfr$ __class__s r r)zBitNetMLP.__init__/s+  )&*B*BH[H[\rc |j|j|j|j||j |z}|S)N) down_projr,act_fn gate_projup_proj)r-xr0s r forwardzBitNetMLP.forward3sFNN4#4#4T[[PQAR5SVZVbVbcdVe5e#fg r)rrrrr)r5 __classcell__r.s@r r#r#.s]|]rr#c2eZdZdedeffd Z d dejdeejejfde ejde e de ejd e e d eeje eje eejffd ZxZS) BitNetAttentionr$ layer_idxcrt|||t|j|j|_yr&)r(r)r hidden_sizer+ attn_sub_norm)r-r$r:r.s r r)zBitNetAttention.__init__9s- +*6+=+=6CVCVWr hidden_statesposition_embeddingsattention_maskpast_key_valuecache_positionkwargsreturnc |jdd}g|d|j}|j|j|j dd} |j |j|j dd} |j |j|j dd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|jjdk7r^|jjdk(r(|jddrtjd nt |jj}||| | | |f|j"sd n |j$|j&d |\}}|j(g|dj+}|j-|}|j/|}||fS) Nrr )sincosrBeagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.g)dropoutscaling)shapehead_dimq_projview transposek_projv_projrupdater:rr$_attn_implementationgetlogger warning_oncer trainingattention_dropoutrMreshape contiguousr=o_proj)r-r>r?r@rArBrC input_shape hidden_shape query_states key_states value_statesrHrG cache_kwargsattention_interface attn_output attn_weightss r r5zBitNetAttention.forward=s$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &S#7 jRUWZ#[ j  %#&snUL'5'<'d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL %  % ! \*k));;;;FFH ((5 kk+. L((r)NN)rrrrintr)torchTensorrrr LongTensorr rr5r6r7s@r r9r98sX|XX+/59 1)||1)#5<<#=>1)!. 1) ! 1) !!1!12 1)-.1) u||Xell3XeELL>Q5RR S1)rr9c eZdZy)BitNetDecoderLayerNrrrr rmrmqr!rrmc eZdZy) BitNetModelNrrrr rorour!rroc2eZdZdgZdZdZdeffd ZxZS)BitNetForCausalLMzlm_head.weightNrDc "t|di|S)a$ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, transformers., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, transformers., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BitNetForCausalLM >>> model = BitNetForCausalLM.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> prompt = f'<|begin_of_text|>User: Hey, are you conscious? Can you talk to me?<|eot_id|>Assistant: ' >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=100) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "User: Hey, are you conscious? Can you talk to me?Assistant: No, I'm not conscious. I'm an artificial intelligence designed to assist with information and tasks. How can I help you today?" ```r)r(r5)r- super_kwargsr.s r r5zBitNetForCausalLM.forward~s4w...r) rrr_tied_weights_keys_tp_plan_pp_planr r5r6r7s@r rqrqys**+HH/ //rrq)rqroBitNetPreTrainedModel)(__doc__typingrrrri cache_utilsrmodeling_flash_attention_utilsrmodeling_outputsr modeling_utilsr processing_utilsr utilsr gemma.modeling_gemmarllama.modeling_llamarrrrrrrconfiguration_bitnetr get_loggerrrXrr#r9rmrorq__all__rrr rs,, B65&+/   H % L 6)n6)r *  * /(/D r