UhN(fdZddlmZddlmZmZmZmZmZm Z m Z ddl Z ddl m cmZddlZ ddl m Z ddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2e*r ddl3m4Z4ddl5m6Z6e+jne8Z9eGddeZ:eGddeZ; d@dZeOeZRgd?ZSy)CzPyTorch Idefics model.) dataclass)AnyCallableDictListOptionalTupleUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs) ModelOutput)ALL_ATTENTION_FUNCTIONSPretrainedConfigPreTrainedModel)Unpack)ALL_LAYERNORM_LAYERS) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) IdeficsConfig)IdeficsPerceiverResampler)IdeficsVisionEmbeddingsIdeficsVisionTransformer) BlockMask)make_flex_block_causal_maskceZdZUdZdZeejed<dZ ee e ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<y)IdeficsBaseModelOutputWithPasta Base class for Idefics model's outputs that may also contain a past key/values (to speed up sequential decoding). Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlast_hidden_statepast_key_values hidden_states attentionsimage_hidden_states)__name__ __module__ __qualname____doc__r'rtorch FloatTensor__annotations__r(r r)r*r+~/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/idefics/modeling_idefics.pyr&r&6s"H6:x 1 129AEOXeE%*;*;$<=>E8B%(9(9":;Br4r&ceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed<y) IdeficsCausalLMOutputWithPasta Base class for Idefics causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlosslogitsr(r)r*r+)r,r-r.r/r8rr0r1r2r9r(rr)r r*r+r3r4r5r7r7cs@)-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8B%(9(9":;Br4r7c tj|jdjddj d|jdj |j }|jd|}|jdd|d<|jdd|d<|jdd|d<|jdd|d<d|vr|d}|jd||d<||jd||d <|d|djd||d<|d|djd||d<||fS|d|djd||d<||fS|d|djd||d<||fS) Nrr pixel_valuesimage_encoder_embeddingsperceiver_embeddingsimage_attention_masktoken_type_idsattention_mask) r0arangeshapeviewrepeattodevice index_selectget) input_ids expand_sizeis_encoder_decoderrAencoder_outputs model_kwargsexpanded_return_idxr@s r5expand_inputs_for_generationrPs  Y__Q'(--b!4;;A{KPPQSTWWXaXhXhi&&q*=>I#/#3#3ND#IL /;/?/?@Z\`/aL+,+7+;+;zfreeze_model..s$]qZ%:$]sTF)r rRrSrTmodulesanyrequires_grad_)modelmodule_exceptionsmappingmmodule_exceptions_mappedrZs @r5 freeze_modelrds\\))\\G 5FFq FF--/) $]D\$]!]  ! !$ '  ! !% ( ) L Gs BcNeZdZdZ ddeeddffd ZdZdefdZ xZ S) IdeficsDecoupledEmbeddinga Implements a decoupling of parameters to allow freezing (or not) a subset of the embeddings. In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `num_additional_embeddings` > 0, then it will create `num_additional_embeddings` additional parameters that are always trained. If `num_additional_embeddings=0`, then the module defaults back to the regular behavior of `nn.Embedding`. Npartially_freezereturnc B|||kDrtd|d|t |d|||||d|||_||_||_||_|r|jjd|j dkDr)tj|j ||||_ yy) a) Args: num_embeddings (`int`): Size of the dictionary of embeddings num_additional_embeddings (`int`): Number of additional embeddings. Only useful when you `partially_freeze=True`. embedding_dim (`int`): The size of each embedding vector partially_freeze: (`bool`, *optional*, defaults to `False`): If `True`, the regular `weight` will be frozen. `additional_weight` is never frozen. padding_idx (`int`, *optional*): The padding index (needs to be less than num_embeddings) Note: there are a lot of other parameters to initialize a standard `nn.Embedding` such as `padding_idx`, `max_norm` or `norm_type`. We are not supporting these. Nz/padding_idx must be within num_embeddings. Got z and )num_embeddings embedding_dimrGdtype padding_idxFr)rjrkrGrlr3) ValueErrorsuper__init__rjrmnum_additional_embeddingsrgweightr^r rTadditional_embedding) selfrjrqrkrgrGrlrmkwargs __class__s r5rpz"IdeficsDecoupledEmbedding.__init__s6  "{^'CN{m[`ao`pqr r  )'#    -&)B& 0  KK & &u -  ) )A -(* #==+ )D % .r4cb|jdk(r tj||jS|j }t j ||jk\}||}|j||jz }d||<tj||j}|||<|S)a we have 2 embeddings, with different indices - one pretrained self.weight and another self.additional_embedding.weight that is being trained. in order to make a lookup of the input ids, we: 1. find out the indices of the entries belonging to the 2nd embedding 2. extract those values while subtracting the size of the first embedding (num_embeddings), since the 2nd embedding starts from 0 and not num_embeddings 3. perform the 2nd embedding lookup 4. now we handle the 1st embedding, we overwrite indices belonging to the 2nd embedding with a padding index 5. perform the 1st embedding lookup 6. now we overwrite the values in the 1st embedding lookup with the values of the 2nd embedding lookup note: for the 1st embedding lookup we could have looked up only the low indices and not do the padding, but then we have to create a new tensor and populate it with 2 tensors that are spread out across various indices - i.e. not a simple concat - I haven't benchmarked the complex case if it's any faster, given that seqlens are usually relatively short it's probably not faster or if faster not by much - but might be a good idea to measure. r) rqF embeddingrrcloner0whererjrs)rtrJadditional_vocab_indicesinput_ids_additional_vocabadditional_embeddings full_vectors r5forwardz!IdeficsDecoupledEmbedding.forwards*  ) )Q .;;y$++6 6OO% #(;;yD 0, then it will create `out_additional_features * in_features` additional parameters that are always trained. If `out_additional_features=0`, then the module defaults back to the regular behavior of `nn.Linear`. N in_features out_featuresout_additional_featuresbiasrgrhc"t||||||||_||_||_||_|r8|j jd|r|jjd|dkDr tj||||||_ yy)aG out_additional_features: int. Number of additional trainable dimensions. Only makes sense when `partially_freeze=True`. partially_freeze: bool. If True, the regular `weight` will be frozen and extra parameters (if any) will be trainable. If False, default to the regular behavior of nn.Linear. Fr)rrrrGrlN) rorprrgrrrrr^rr rS additional_fc) rtrrrrrgrGrlrvs r5rpzIdeficsDecoupledLinear.__init__@s lD&%H'>$ 0&(  KK & &u - ((/ "Q &!#'4 "D  'r4inputctj||j|j}|jdkDr)|j |}t j||fd}|S)Nrr;)rxlinearrrrrrr0cat)rtroutputadditional_featuress r5rzIdeficsDecoupledLinear.forwarddsV%dii8  ' '! +"&"4"4U"; YY(;>r4>P% Ht?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r4c^t|jjd|jS)Nz, eps=)tuplerrrCrrs r5rzIdeficsRMSNorm.extra_reprs* ))*+6$2G2G1HIIr4)gư>)r,r-r.rprrrrs@r5rrys$+Jr4rc.eZdZdfd ZdZddZxZS)IdeficsEmbeddingct|||_||_||_d|jt j d|jdt jj|t j|jz zz }|jd|d|j||jjt j y) N?rrrlrGrlinv_freqF persistentseq_lenrGrl)rorpdimmax_position_embeddingsbaser0rBint64rFfloatregister_buffer_set_cos_sin_cacherrGget_default_dtype)rtrrrrGrrvs r5rpzIdeficsEmbedding.__init__s '>$  II Q!5;;?BB&X]XcXcBdgkgogoo q  ZeD +DMM4H4HPUPgPgPi  r4c||_tj|j|tjj |j }tj d||j }tj||fd}|jd|jj|d|jd|jj|dy) Nrzi,j->ijr;r cos_cachedFr sin_cached) max_seq_len_cachedr0rBrtype_asreinsumrrcosrFsin)rtrrGrlrYfreqsembs r5rz#IdeficsEmbedding._set_cos_sin_caches") LL00u{{ S [ [\`\i\i j Y4==9iiB/ \3779<<+>5Q \3779<<+>5Qr4c||jkDr(|j||j|j|jd|j |j|j d|j |jfS)Nrr)rrrGrlrrFr)rtxrs r5rzIdeficsEmbedding.forwardsy T,, ,  # #GAHHAGG # T OOHW % ( (qww ( 7 OOHW % ( (qww ( 7  r4)ii'NrV)r,r-r.rprrrrs@r5rrs "R r4rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr;rr)rCr0r)rx1x2s r5 rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r4c||j|}||j|}||zt||zz}||zt||zz}||fS)anApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`): The position indices of the tokens corresponding to the query and key tensors. For example, this can be used to pass offsetted position ids when working with a KV-cache. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrr position_ids unsqueeze_dimq_embedk_embeds r5apply_rotary_pos_embrsg* l  % %m 4C l  % %m 4C3w;q>C/0G3w;q>C/0G G r4c2eZdZdededeffd ZdZxZS) IdeficsMLPrintermediate_size hidden_actct|tj||d|_tj||d|_tj||d|_t||_y)NFr) rorpr rS gate_proj down_projup_projr act_fn)rtrrrrvs r5rpzIdeficsMLP.__init__s[ ;0AN#4kNyy.?eL Z( r4c|j|j|j||j|zSrV)rrrr)rtrs r5rzIdeficsMLP.forwards0~~dkk$..*;>SWS`S`akklmopq !]388eT^^UYUbUbcmmnoqrsJ;;}5::3t~~W[WdWdeoopqstuL+002LAvq%56;;CY]YfYfgqqrsuvwJ ,-223PTP]P]^hhijlmn  %%b)  % .+ +J!|SU=STHC';L*VY[^`l'm $L*  %,n=L'5'<'O##L '>dkk>^>^&_#$7      % $}}C$,,LL %  % ! \"))#ub9DDF kk+. LL.88r4)rFNFNNNNNFFN)r,r-r.r/rrrrrrpr0rr LongTensorr rrrs@r5rr svG #(#'$#'OWOWOW OW ! OW ! OWOWC=OWbeU\\eCece 4815378<"'59J9||J9#5<<0J9!. J9 u//0 J9 !u||!45 J9 J9J9!!1!12J9 u||Xell3XeELL>Q5RR SJ9r4rcJeZdZd dedeeffd Z ddejdeejdeejdee ejdee d ee d eejd e ejee ejejfffd Z xZS)IdeficsDecoderLayerrrct||j|_t|j|j|j |||_t|j|j|j|_ t|j|j|_ t|j|j|_|j |_y)N)rrrrrrrrr)rorprrnum_attention_headsr self_attnrrrmlprrinput_layernormpost_attention_layernormrtrrrvs r5rpzIdeficsDecoderLayer.__init__s !--)((00NN  (($66((  .f.@.@fFYFYZ(6v7I7IvObOb(c%~~ r4r)rArrrrrrhc |} |j|}|jd|||||||d|\}} } tjj ||j|j }| |z}|} |j |}|j|}tjj ||j|j }| |z}|f} |r| | fz } |r| | fz } | S)a] Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states )r)rArrrrrrr3)r6r4r rrrr7r5) rtr)rArrrrrruresidualself_attn_weightspresent_key_valueoutputss r5rzIdeficsDecoderLayer.forwards2!,,]; ?Mdnn ? ')%)/) ?  ? ; (*; --mt||VZVcVc-d  =0 !55mD /  --mt||VZVcVc-d  =0  "  )+ +G  )+ +Gr4rV)NNNFFN)r,r-r.rrrrpr0rr.r rr1rrrs@r5r0r0s&}&#&,26378<,1$)59:||:!.:u//0 : !u||!45 : $D> :D>:!!1!12: u  (51B1BEDUDU1U+V"WW X:r4r0cjeZdZddedeeffd Z ddejdeejdeejdeejdeejd ee d ee d ee ejd e ejee ejejfffd Z xZ S)IdeficsGatedCrossAttentionLayerrrc  t||j|_t|j|jd|j ||j ||_t|j|j|j|_ t|j|j|_t|j|j|_|j |_t#j$|_t#j$|_|j*dk(r|j,dk(rtt#j.t1j2dd|j|_t#j.t1j2dd|j|_n|j,dk(r\t#j.t1j2d|_t#j.t1j2d|_nt9d |j,d |j*d k(r|j,dk(rtt#j.t1j:dd|j|_t#j.t1j:dd|j|_n|j,dk(r\t#j.t1j:d|_t#j.t1j:d|_n|t9d |j,d |j*d vr;|j,dk(rt#j.t1j<d |j>dd|jf|_t#j.t1j<d |j>dd|jf|_n|j,dk(rut#j.t1j<d |j>d|_t#j.t1j<d |j>d|_n2t9d |j,d tAd|j*dtC|dr tC|ds t9dy)NT)rrrrrrrr2rzerosvectorrrz Unknown value for `alpha_type` ()r>normalrandomgaussianr)rstdrzAlpha initialization scheme z not yet implemented!alpha_cross_attn alpha_densez+Alpha parameters not initialized correctly!)"rorprrr3rr cross_attnrrrr5rrr6r7rr Tanhact_cross_attn act_densealpha_initializer alpha_typerr0rArHrIrnrrDalphas_initializer_rangeNotImplementedErrorrr8s r5rpz(IdeficsGatedCrossAttentionLayer.__init__s !--*((00#NN!00 (($66((  .f.@.@fFYFYZ(6v7I7IvObOb(c%nn  ggi  # #w .  H,(* U[[AtGWGW5X(Y%#%<< Aq$BRBR0S#T ""g-(* U[[^(D%#%<< A#?  #CFDUDUCVVW!XYY  % % /  H,(* UZZ1dFVFV5W(X%#%<< 1aAQAQ0R#S ""g-(* UZZ](C%#%<< 1 #>  #CFDUDUCVVW!XYY  % %)I I  H,(* LLcv/N/NVWYZ\`\l\lUmn)%$&<<LLcv/N/NVWYZ\`\l\lUmn$ ""g-(* LLcv/N/NVWY)%$&<< #6KjKjrs0u#v  #CFDUDUCVVW!XYY&(DVE]E]D^^s&tu u01gdM6RJK K7Sr4r)rAr+r?cross_attention_gaterrrrhc  | td| td| td|} |j|}|jd ||||d| \}} } tj j ||j|j}|j|dk(dddddfd}| |j|j|zz}|} |j|}|j|}tj j ||j|j}| |j|j|zz}|f} |r| | fz } |r| | fz } | S) a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. image_attention_mask (`torch.FloatTensor`, *optional*): image attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. cross_attention_gate (`torch.FloatTensor`, *optional*): gate of size `(batch, seq_len)` used to zero-out cross-attention output for tokens attending no images. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states Nzt`image_hidden_states` is required for Idefics cross attention module which are visual features to be conditioned on.z`cross_attention_gate` is required for Idefics cross attention module to zero-out the cross-attention hidden_states attending to no images.zMPast key value states are not implemented for Idefics cross attention module.)r)rrArrrrr3)rnrQr6rJr rrrr masked_fillrLrHr7r5rMrI)rtr)rAr+r?rRrrrrur:r;r<r=s r5rz'IdeficsGatedCrossAttentionLayer.forwardBs:  &#  '^   %%&uv v ,,]; ?Ndoo? '0// ?  ? ; (*; --mt{{UYUbUb-c %113G13LaQRTXj2Y[^_  4#6#6t7L7L#MP]#]] !55mD /  --mt{{UYUbUb-c  4>>$2B2B#Cm#SS  "  )+ +G  )+ +Gr4rVr-)r,r-r.rrrrpr0rrr r1rrrs@r5r?r?s@L}@L#@LJ266:7;7;,1$)8<K||K!.K&ell3 K 'u||4 K 'u||4 K$D>KD>K!u||!45K u  (51B1BEDUDU1U+V"WW XKr4r?c:eZdZeZdZdZddgZdZdZ dZ dZ dZ dZ y)IdeficsPreTrainedModelr_Tr0r?Fc |jj}t|tjtj frY|j jjd||j%|jjjyyt|tjrf|j jjd||j2|j j|jjyyt|tjrJ|j jjd|jjjyt|tr&|j jjdyt|t r%|j"jjyt|t$rV|jj&dk(rI|j(jj|j*jjy|jj&dk(rK|j(jjd|j*jjdy|jj&dvrw|j(jjd|jj,|j*jjd|jj,yyt|t.r%|j0jjyy)Nr)rrGrrAr>rDrErF)rinitializer_rangerWr rSConv2drrdatanormal_rzero_rTrmrRfill_rr!class_embeddingr?rNrHrIrPr latents)rtrZrGs r5 _init_weightsz$IdeficsPreTrainedModel._init_weightsspkk++ fryy"))4 5 MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  - MM   $ $S ) KK   " " $  / MM   $ $S )  7 8  " " ' ' / / 1  ? @{{,,7'',,224""''--/..&8'',,2237""''--c2..2RR'',,44#4;;CgCg4h""''//Sdkk>b>b/cS 9 : NN   ' ' );r4N)r,r-r.r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_cache_class_supports_flash_attn_2_supports_static_cache_supports_attention_backendr`r3r4r5rVrVsA L&*#.0QRN !""&*r4rVc eZdZy)KwargsForCausalLMN)r,r-r.r3r4r5rkrksr4rkc'eZdZdZdeffd Zd#dZgfdZgfdZdZ dZ e e d$d e ejd e ej d e ejd e eej$d e ej$de ej$de ej$de ej$de ej de ede ede ede ede ede ejdeedeeeff"dZ d%d eej dfdej dej d edef dZed ej deded ej<dej d!ef d"ZxZ S)& IdeficsModelz Transformer decoder consisting of `config.num_hidden_layers` layers. Each layer is a [`IdeficsDecoderLayer`] Args: config: IdeficsConfig rc jt||||_|j|_|j |_t |j |j|j|j|j|_ |jj|_ |j|_ t|j|_|jr]|j }t#||jj$|j&|j(|j*|j,|_t1j2t5|j6Dcgc]}t9||c}|_|j<|_|j6|j<z}t1j2t5|Dcgc]}t?||c}|_ d|_!tE|j|jF|_$|jK|jM|ycc}wcc}w)N)rjrqrkrgrm)rFr)'rorpr pad_token_idrm vocab_sizerfadditional_vocab_sizerfreeze_text_layers embed_tokensr image_sizer" vision_model use_resamplerperceiver_configr rresampler_depthresampler_n_headsresampler_head_dimresampler_n_latentsperceiver_resamplerr ModuleListrangenum_hidden_layersr0layerscross_layer_intervalr?gated_cross_attn_layersgradient_checkpointingrrnorm post_initfreeze_relevant_params)rtrrwinum_cross_layersrvs r5rpzIdeficsModel.__init__s   !.. ++5!,,&,&B&B ,,#66((  !..99#114V5I5IJ   %66 '@$$.. 00 22 33 44 (D $mm?DVE]E]?^ _! 1 5 _  %+$?$?!!33t7P7PP')}}KPQaKb ca ,Vq A c( $',#"6#5#56;N;NO   ##F+ ` ds H+<H0c| |j}|jr|j|j|jr"t |j |j yyN)r`)rrrfreeze_text_module_exceptionsfreeze_vision_layersrdrufreeze_vision_module_exceptions)rtrs r5rz#IdeficsModel.freeze_relevant_paramssQ >[[F  $ $  # #F$H$H I  & & **f>d>d e 'r4cX|j|jfD]}t||yr)rrrd)rtr`rZs r5rrzIdeficsModel.freeze_text_layerss+{{DII. FF 3D E Fr4c2t|j|yr)rdru)rtr`s r5rz!IdeficsModel.freeze_vision_layerssT&&:KLr4c|jSrVrsrs r5get_input_embeddingsz!IdeficsModel.get_input_embeddings s   r4c||_yrVrrtrs r5set_input_embeddingsz!IdeficsModel.set_input_embeddingss !r4rJrArr( inputs_embedsr<r=r>r?rroutput_hidden_statesinterpolate_pos_encoding return_dictrrurhc | |jn |j}| | n|jj} | | n|jj} | | n|jj} ||n|jj }|du|duz r t d|jr%|jr| rtjdd} ||j|}d}| rIt|ts9d}| t}n*tj|}tjd|j \}}}||j#nd}||z}|2t%j&|||j dz|j }|F|D|j)j+d dz }|j-|dk(d|dd| df}n||j/d}t1|||fDcgc]}|duc}d k7r t d |~|j3|j4| }|j dd \}}|j7j8||zg|j d d}|j;|| j<}nJ|H|j?\}}}}|j3|j4| }|j9||z||}|jj@rN|4|jC}|j?d|j?d }}n|j?\}}}}|}n0|#j?d|j?d }}n t d|j9||z|}| j?d}| j/d } | jEddd|} | j9||||z} |C|j?\}}}||f} | t%jF| | } |jI| } nd} | dk(jKd j3|j4jMdj3|}!|2t%jF||ft$jN|j }|jQ||||| }|}"| rdnd}#| rdnd}$d}%tS|jTD]\}&}'| r|#|"fz }#fd}(|jr[|jrOd}| rtjdd} |jW|(|'|"||||| |!| | |&|jX|jZ|})n+|(|'|"f||||| |!| | |&|jX|jZ|d })|)d}"| r |)| rd nd}%| s|$|)dfz }$|j]|"}"| r|#|"fz }#| r|%nd}*|r|*j_}*|j9||||}ta|"|*|#|$|Scc}w)ab image_encoder_embeddings (`torch.FloatTensor`, *optional*): The output of the image encoder. perceiver_embeddings (`torch.FloatTensor`, *optional*): The output of the perceiver resampler. image_attention_mask (`torch.LongTensor`, *optional*): The attention mask for the image encoder. Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)rrrGr;rz_Exactly 1 of pixel_values, image_encoder_embeddings or perceiver_embeddings has to be not-None.)rlrG)r<rzBIf `perceiver_embeddings` are passed, use_resampler should be Truerrrr3c t| | zdk(r| | | z}||f|||||| dd}|d}||f||||| | d}|S)Nr)rAr+r?rRrrr)rArrrrrr3) main_blockr)rArrr+r?rRrrrrrrxblockr= layer_outputsrus r5vblockz$IdeficsModel.forward..vblocks"33q84YBV5VWF$% '5,?-A-A*;"+'+ ! G%,AJM *! !#1!-#1&7'#1 ! ! %$r4zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...) rArrr+r?rRrrrrrr)r'r(r)r*r+)1rGrrrruse_return_dictrnrrrrrsrWrrfrom_legacy_cacherCget_seq_lengthr0rBlongcumsum masked_fill_rsumrFrlrrDrur'rrvr|rErinvert_attention_maskr]squeezer_update_causal_mask enumerater_gradient_checkpointing_funcrrrto_legacy_cacher&)+rtrJrArr(rr<r=r>r?rrrrrrrurGreturn_legacy_cache batch_size seq_lengthr%past_key_values_lengthseq_length_with_pastr num_imagesr+ image_seq_lenimage_hidden_size text_seq_lenimage_batch_sizeimage_sequence_lengthimage_hidden_shaperRr)all_hidden_statesall_self_attnsnext_decoder_cacheidx decoder_layerrr next_caches+ ` r5rzIdeficsModel.forwards:&/%:!! @T@T1B1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKATATUVAW(W`m`t`tN  %,*>)..077;a?L  % %n&91 ='J;<8L  !)33A6L L2JL`#abaT b cgh hq  %'??F?KL%1%7%7%; "J 9<22499*z:QkT`TfTfghgiTjkL#'"3"3)D\#4# & 1G_GdGdGf DJ M3D":"="=DJJW]"="^ "5":":: ;RTact"u  ;; $ $#+'+'?'?@S'T$3G3L3LQ3OQeQjQjklQm0 K_KdKdKfH J 7H"6  ! )/B/G/G/JL_LdLdefLg,Mab b166z:P]C]_pq,003 3==bA3::1aMR388\S]`mSmn  *9L9Q9Q9S 6 3Q"24I!J #+',zz2DV'T$#'#=#=>R#S #' $83#>"C"C"C"K!O!OVZV`V`!O!a j jop j quu    !"ZZ12%**]MaMaN11 M>?L] & #7BD0d!"+DKK"8c 6 C#!m%55!+ %Z**t}}"&''t!&I $ A A!!" #'((%--00"! $!'!!!$2!-#2(;)=)=&7'!)-)B)B,0,H,H#1!! $*!,M%28I1q%Q" =#3"55Gc 6J -0  -!1 1 +4'$ #335J166z:}^op-+&+% 3   ycs- W(r# input_tensorc |jjdk(r||dk(jr|Sy|jjdk(r't|tj r t |}|S||jnd}| |jnd}|jjdk(r(|s&|s$tj||||jry|j}|jd} |r|j} n1t|tj r|jd n|| zdz} |j|| | |||jd } |jjdk(rQ|O|j j"d vr7|s5t j$|j&} tj(| | } | S) Nflash_attention_2rflex_attentionrFr)rr is_trainingrr;)sequence_length target_lengthrlrr)cudaxpunpu)rr"r]rWr0rr$ris_compileabler_ignore_causal_mask_sdparrlrCget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrGtypefinfomin_unmask_unattended) rtrArrr(rpast_seen_tokensusing_compilable_cacherlrr causal_mask min_dtypes r5rz IdeficsModel._update_causal_mask(s ;; + +/B B)~/D.I.I.K%% ;; + +/? ?.%,,7!<^!L! ! @O?Z?99;`aCRC^!?!?di ;; + +v 5>T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCKQZ[Kr4rrrlrc ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerlrGr)diagonalrr;r)rr0rrfullrGtriurBr#expandrzrCrFrT) rArrrlrrrurr mask_length padding_masks r5rzBIdeficsModel._prepare_4d_causal_attention_mask_with_cache_positionls>  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r4rV)NNNNNNNNNNNNFNNF)!r,r-r.r/rrprrrrrrrrrr0r.rrr1rrrr r r&rrr staticmethodrrlrrrs@r5rmrms.,}.,`f46F68M!"151537=A5948@D<@7;$(,0/338&*59!R E,,-R !.R u//0 R "$u'8'8"9: R   1 12 R u001R #+5+<+<"=R 'u'8'89R 'u||4R D>R $D>R 'tnR #+4.R d^R !!1!12!R "-.#R $ u44 5%R R v#( BellK78BllB B  B  BH4 444{{ 4  4  44r4rmc)eZdZddgZd$fd ZdZdZdZdZdZ d Z d Z e e d%d eej d eej"d eej deeej&deej&deej&deej&deej&deej"deej deedeedeedeedeedeej deedeeeff$dZ d&fd Z d'ded eeefd!edeeefffd" Z e!d#Z"xZ#S)(IdeficsForVisionText2Textzmodel.embed_tokens.weightzlm_head.weightct||t||_t |j |j |jd|j|_ |jy)NF)rrrrrg) rorprmr_rrrprqfreeze_lm_headlm_headr)rtrrurvs r5rpz"IdeficsForVisionText2Text.__init__s[  !&) -****$*$@$@#22    r4c.|jjSrVr_rsrs r5rz.IdeficsForVisionText2Text.get_input_embeddingsszz&&&r4c&||j_yrVrrs r5rz.IdeficsForVisionText2Text.set_input_embeddingss"' r4c|jSrVrrs r5get_output_embeddingsz/IdeficsForVisionText2Text.get_output_embeddingss ||r4c||_yrVr)rtnew_embeddingss r5set_output_embeddingsz/IdeficsForVisionText2Text.set_output_embeddingss % r4c||_yrVr_)rtdecoders r5 set_decoderz%IdeficsForVisionText2Text.set_decoders  r4c|jSrVrrs r5 get_decoderz%IdeficsForVisionText2Text.get_decoders zzr4c|j}|j}t|jddr`|j|_|j dkDr@|j |j k(sJ|jj|j_t|drJt|dr=|j|_ t|drt|dr|j |_yyyyy) z Overwrite `transformers.modeling_utils.PreTrainedModel.tie_weights` to handle the case of IdeficsDecoupledLinear and IdeficsDecoupledEmbedding. tie_word_embeddingsTrrrjrrqN) rrgetattrrrrrqrrsrrrjr)rtoutput_embeddingsinput_embeddingss r5 tie_weightsz%IdeficsForVisionText2Text.tie_weightss !668446 4;; 5t <'7'>'>  $99A=(@@DTDnDnnnn9I9^9^9e9e!//6 $n 5'BRTd:e-=-L-L  *(*CD "=J=Mr?labelsrrrrrrrurhc | | n|jj} | | n|jj} ||n|jj}|jd||||||||| | | | |d|d|}|d}|j |}d}| *|j d|| |jjd|}t|||j|j|j|jS)aC labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_encoder_embeddings (`torch.FloatTensor`, *optional*): The output of the image encoder. perceiver_embeddings (`torch.FloatTensor`, *optional*): The output of the perceiver resampler. image_attention_mask (`torch.LongTensor`, *optional*): The attention mask for the image encoder. Example: ```python >>> from transformers import AutoProcessor, IdeficsForVisionText2Text >>> model = IdeficsForVisionText2Text.from_pretrained("HuggingFaceM4/idefics-9b") >>> processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics-9b") >>> dogs_image_url_1 = "https://huggingface.co/datasets/hf-internal-testing/fixtures_nlvr2/raw/main/image1.jpeg" >>> dogs_image_url_2 = "https://huggingface.co/datasets/hf-internal-testing/fixtures_nlvr2/raw/main/image2.jpeg" >>> prompts = [ ... [ ... "User:", ... dogs_image_url_1, ... "Describe this image.\nAssistant: An image of two dogs.\n", ... "User:", ... dogs_image_url_2, ... "Describe this image.\nAssistant:", ... ] ... ] >>> inputs = processor(prompts, return_tensors="pt") >>> generate_ids = model.generate(**inputs, max_new_tokens=6) >>> processor.batch_decode(generate_ids, skip_special_tokens=True) ```NT)rJrArr(rr<r=r>r?rrrrrrr)r9rrp)r8r9r(r)r*r+r3) rrrrr_r loss_functionrpr7r(r)r*r+)rtrJrArr(rr<r=r>r?rrrrrrrrur=r)r9r8s r5rz!IdeficsForVisionText2Text.forwards%x2C1N-TXT_T_TqTq$8$D $++JjJj &1%r=r<rF)r(rArrrrr?rJrr?)rrvpoproprepare_inputs_for_generationrC)rtrJrArrr(rr<r+r?rru images_kwargs model_inputsrrvs r5rz7IdeficsForVisionText2Text.prepare_inputs_for_generationDs   *{{((8K 45.s$gjZ%<%b $D>b 'tnb #+4.b d^!b "!!1!12#b $*+%b & u33 4'b b N !+b$) 38n!  c3h 4r4r)rrmrV)rFNN)r)r)Tr/ dataclassesrtypingrrrrrr r r0torch.nn.functionalr rrxtorch.utils.checkpoint activationsr cache_utilsrr generationrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_outputsrmodeling_utilsrrrprocessing_utilsr pytorch_utilsrutilsrrrrrconfiguration_ideficsr perceiverr visionr!r"!torch.nn.attention.flex_attentionr#integrations.flex_attentionr$ get_loggerr,rr&r7rPrdrTrfrSrModulerappendrrrrrrrrr0r?rVrkrmr__all__r3r4r5r&sK(!DDD !.)>B+XX&1hh00E !;J   H % )C[)C )CX &CK&C &CV *#Z+- k  k \> RYY> DJRYYJ.N+$ uxx$ N(:PP2% II% <<% % << % U\\* %  %%0a9ryya9JN"))NbNbiiNb**_****Z?,j>`)``Fk 6k\ Rr4