Uh FddlmZmZmZmZmZddlZddlmcm Z ddlmZddl m cm cmZddlmZddlmZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd l m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,e*rddl-m.Z.ddl/m0Z0m1Z1ndZ.e)r ddl2m3Z3m4Z4nd\Z4Z3e&r ddl5m6Z6ddl7m8Z8e'jre:Z;dZeZSe$Gd?d@e"ZTGdAdBejZUe$GdCdDeTZV dMdEeej|eej|dfdFee?d!eej|deej|e?ffdGZWGdHdIeTeZXgdJZYy)N)CallableListOptionalTupleUnionN)nn)ACT2FN)Cache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)auto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)is_causal_conv1d_availableis_mamba_2_ssm_available)GraniteMoeHybridConfig)selective_state_update)mamba_chunk_scan_combined mamba_split_conv1d_scan_combined)causal_conv1d_fncausal_conv1d_updateNN) BlockMask)make_flex_block_causal_maskc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2s /var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/granitemoehybrid/modeling_granitemoehybrid.py rotate_halfr2@sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer2)qkcossin position_ids unsqueeze_dimq_embedk_embeds r1apply_rotary_pos_embr>GsY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r3 hidden_statesn_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r+expandreshape)r?r@batchnum_key_value_headsslenhead_dims r1 repeat_kvrIbso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr3modulequerykeyvalueattention_maskscalingdropoutc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr(r r')r*dtype)ptrainingr)rInum_key_value_groupsr,matmul transposer+r functionalsoftmaxfloat32torSrPrU contiguous) rJrKrLrMrNrOrPkwargs key_states value_states attn_weights causal_mask attn_outputs r1eager_attention_forwardrdns3 ; ;updaterhrdrg_attn_implementationgetrnrorrUrrrOr~)rr?rNr:rrrrr^bszq_len_ query_statesr_r`r8r9 cache_kwargsattention_interfacercras r1forwardz!GraniteMoeHybridAttention.forwards &**, UA{{=1 [[/ {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm *=*I&|S  *';L*VY[^'_ $L*  %#&snUL'5'<'d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL %  % ! \"&&sE26 kk+. L.88r3)NNNFNN)rq __module__ __qualname____doc__rintrmr,Tensorr LongTensorr boolrr __classcell__rps@r1rfrfsG`5`#`F2637*.59KO69||69!.69u//0 69 ! 69  69!!1!1269&eELL%,,,F&GH69 u||Xell3XeELL>Q5RR S69r3rfcBeZdZdZej dfdeffd ZxZS) HybridMambaAttentionDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. Nrgc Rt ||||||j|_d|_|j}|j }g|_g|_g|_t|jD]*}|j|dk(r|xj tj||j|jzd|jz|zz|||gz c_|xjtj||j |j"|||gz c_|xj tj$gg|z|gz c_|xjtj$gg|z|gz c_|jj'|-t|jDcgc]}tj$gg|z|c}|_t|jDcgc]}tj$gg|z|c}|_ycc}wcc}w)NFmambar(devicerSr)rlrmlayers_block_typehas_previous_state mamba_d_conv mamba_d_state conv_states ssm_statestransformer_layersrangenum_hidden_layersr,zeros mamba_expandrsmamba_n_groups mamba_n_heads mamba_d_headtensorappend key_cache value_cache) rrg batch_sizerSrconv_kernel_sizessm_state_sizeirrps r1rmz)HybridMambaAttentionDynamicCache.__init__s UF;!'!9!9"'!..--"$v//0 2A%%a(G3  KK",,v/A/AAAH]H]D]`nDnn(%# % KK",,++&%#  $   U\\2$2CF%S$TT ELL" 1B6$R#SS''..q11 24SXX^XpXpRqrQ%,,tj'8HrTYZ`ZrZrTstqELL" ):6Jtsts 3"H4"H$) rqrrrr,float16rrmrrs@r1rrs+ JO_c%u5%u%ur3r input_tensorpad_sizect|jdk(r ddddd|ddfnddd|ddf}tjjj ||ddS)z Padding x tensor with `pad_size` on the seq_len dim (dim=1) Assumes that we only have tensors of either size 4 or 3 rconstant)moderM)lenr+r,rrYpad)rr pad_shapes r1pad_tensor_by_sizer!sf 47|7I7I3Ja3OAq!Q!Q/VWYZ\]_gijlmUnI 88   " "<ST " UUr3ct||}t|jdk(r.|j|jdd||jdS|j|jdd||jd|jdS)z Padding input_tensor with `pad_size` on the seq_len dim (dim=1) and simultaneously splitting it into chunk sequences. Assumes that we only have tensors of either size 4 or 3 r rr'r()rrr+rD)rr chunk_sizes r1reshape_into_chunksr,s&lH=L <  !###L$6$6q$92z