Uh^ddlmZmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z mZddlmZmZddlmZdd lmZmZd d lmZmZmZmZmZmZmZd d lm Z d d l!m"Z"m#Z#ejHe%Z&Gdde Z'GddeZ(Gdde"Z)GddeZ*GddeZ+GddeZ,GddeZ-GddeZ.eGddeZ/Gd d!eZ0gd"Z1y)#)CallableOptionalTupleUnionN)PretrainedConfig)FlashAttentionKwargs)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging)CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayerCLIPVisionEmbeddingsCLIPVisionModelCLIPVisionTransformer)eager_attention_forward)VisionRotaryEmbeddingapply_rotary_pos_emb_visioncFeZdZdZdZdZ dfd ZxZS)MLCDVisionConfiga This is the configuration class to store the configuration of a [`MLCDVisionModel`]. It is used to instantiate a MLCD vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the vision encoder of the MLCD [DeepGlint-AI/mlcd-vit-bigG-patch14-336](https://huggingface.co/DeepGlint-AI/mlcd-vit-bigG-patch14-336) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 1664): Dimensionality of the encoder layers and the pooler layer. intermediate_size (`int`, *optional*, defaults to 8192): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. projection_dim (`int`, *optional*, defaults to 1024): Dimensionality of text and vision projection layers. num_hidden_layers (`int`, *optional*, defaults to 48): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. num_channels (`int`, *optional*, defaults to 3): The number of input channels. image_size (`int`, *optional*, defaults to 336): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to 14): The size (resolution) of each patch. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. layer_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the layer normalization layers. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_factor (`float`, *optional*, defaults to 1.0): A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing). Example: ```python >>> from transformers import MLCDVisionConfig, MLCDVisionModel >>> # Initializing a MLCDVisionConfig with DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> configuration = MLCDVisionConfig() >>> # Initializing a MLCDVisionModel (with random weights) from the DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> model = MLCDVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mlcd_vision_model vision_configc t|di|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_y)N)super__init__ hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_groups num_channels patch_size image_sizeinitializer_rangeinitializer_factorattention_dropoutlayer_norm_eps hidden_act)selfr$r%r&r'r(r)r+r*r0r/r.r,r-kwargs __class__s w/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/mlcd/modular_mlcd.pyr#zMLCDVisionConfig.__init__dsz" "6"&!2!2#6 $8!($$!2"4!2,$) ii 0riPgelugh㈵>g{Gz??)__name__ __module__ __qualname____doc__ model_typebase_config_keyr# __classcell__r3s@r4rr*sH4 l%J%O%%r5rc eZdZy)MLCDMLPN)r=r>r?r!r5r4rFrFsr5rFc4eZdZdededej fdZy)MLCDRotaryEmbeddingnum_patches_heightnum_patches_widthreturnctj||jjj dj d|}tj||jjj dj |d}tj |j|jgd}t||}tj||jj|jj}tj||j}||jd} | S)a} Calculate the Rotary Position Embedding (RoPE) for MLCDVisionModel based on the grid size. Args: num_patches_height (int): Number of patches in the height dimension. num_patches_width (int): Number of patches in the width dimension. Returns: torch.Tensor: Rotary positional embeddings for the given grid size. )devicer8rdim)rMdtype) torcharangeinv_freqrM unsqueezeexpandstackflattenmaxrQouter) r1rIrJhpos_idswpos_idspos_ids max_grid_sizeseqrotary_pos_emb_fullrotary_pos_embs r4forwardzMLCDRotaryEmbedding.forwards  LL+DMM4H4H I S STU V ] ]^`bs t  LL*4==3G3G H R RST U \ \]oqs t  ++x//183C3C3EFBO.0AB ll=1E1ET]]M`M`a#kk#t}}=-W5==a@r5N)r=r>r?intrRTensorrbr!r5r4rHrHs ##%,,r5rHc\eZdZdeffd Zdej dejfdZxZ S)MLCDVisionEmbeddingsconfigc(t|||`yN)r"r#position_embeddingr1rgr3s r4r#zMLCDVisionEmbeddings.__init__s   #r5 pixel_valuesrKcT|jd}|jjj}|j|j |}|j dj dd}|jj|dd}tj||gd}|S)Nr)rQrr8rNrO) shapepatch_embeddingweightrQtorX transposeclass_embeddingrVrRcat)r1rl batch_size target_dtype patch_embeds class_embeds embeddingss r4rbzMLCDVisionEmbeddings.forwards!''* ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C r5) r=r>r?rr#rR FloatTensorrdrbrCrDs@r4rfrfs-$/$ E$5$5 %,, r5rfceZdZdZdeffd Z d dejdeejejfde ejde e deeje ejff d Z xZ S) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://arxiv.org/abs/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://arxiv.org/abs/2104.09864 rgcTt|||j|_d|_yNF)r"r#r( is_causalrks r4r#zMLCDAttention.__init__s%  $*$?$?!r5 hidden_statesposition_embeddingsattention_maskr2rKc <|jdd\}}|j|j|||j|jf}|j |j|||j|jf}|j |j|||j|jf} |djdj} |djdj} t||| | \}}|jddddj}|jddddj}| jddddj} t} |jjdk7r^|jjdk(r(|jdd rt j#d nt$|jj} | |||| |f|j&sd n |j(|j*|j,d |\} }| jddddj} | j/||d} |j1| } | jdddj} | |fS) NrNrr8rreagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r;)dropoutscalingr)rnq_projreshape num_headshead_dimk_projv_projrUfloatrpermute contiguousrrg_attn_implementationgetlogger warning_oncer trainingrscalerviewout_proj)r1rrrr2ru seq_length query_states key_states value_statescossinattention_interface attn_output attn_weightss r4rbzMLCDAttention.forwards|"/!4!4Sb!9 J{{=199:zSWSaSacgcpcp:qr [[/77ZQUQ_Q_aeanan8op {{=199:zSWSaSacgcpcp:qr "!$..q1779!!$..q1779#>|ZY\^a#b j$++Aq!Q7BBD ''1a3>>@ #++Aq!Q7BBD (? ;; + +w 6{{//69fjjI\^c>d##L '>dkk>^>^&_#$7      % $}}C$,,JJnn %  % ! \"))!Q15@@B !&&z:rB mmK0 !))!Q2==? L((r5ri)r=r>r?r@rr#rRrdrrrr rbrCrDs@r4r|r|s/26 2)||2)#5<<#=>2)!. 2) -. 2) u||Xell33 4 2)r5r|c eZdZdeffd Z d dej deej ej fdeej dee deejf dZ xZ S) MLCDEncoderLayerrgcDt||t||_yri)r"r#r| self_attnrks r4r#zMLCDEncoderLayer.__init__s  &v.r5rrrrrKc|}|j|}|j||||\}}||z}|}|j|}|j|}||z}|f}|r||fz }|S)a Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`Tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. rrrr) layer_norm1r layer_norm2mlp)r1rrrrresidualroutputss r4rbzMLCDEncoderLayer.forwards*!((7 &*nn' 3)/ '5' # | !=0  ((7 /  =0  "    &Gr5r~) r=r>r?rr#rRrdrrboolrzrbrCrDs@r4rrs{///26,1 *||*#5<<#=>*!. * $D> * u  ! *r5rceZdZdZdeffd Z d dejdeejejfde ejde e de e d e e d e ee ffd ZxZS) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig rgc$t||y)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.N)r"r#rks r4r#zMLCDEncoder.__init__;s  r5 inputs_embedsrrroutput_hidden_states return_dictrKc||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|} t |j D]b\} } |r|| fz}|j r,|jr |j| j| |||} n | | |||} | d} |sZ|| dfz}d|r|| fz}|std| ||fDSt| ||S)aj Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`Tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr!rrr8c3&K|] }|| ywrir!).0vs r4 z&MLCDEncoder.forward..seqWXWdes)last_hidden_stater attentions) rgruse_return_dictr enumeratelayersgradient_checkpointingr_gradient_checkpointing_func__call__tupler ) r1rrrrrrencoder_statesall_attentionsridx encoder_layer layer_outputss r4rbzMLCDEncoder.forward?sHD%9$D $++JjJj &1%>Ne]NN$See e+(%  r5NNNN)r=r>r?r@rr#rRrzrrdrrrr rbrCrDs@r4rr2s!/!26,0/3&*L ((L #5<<#=>L !. L $D> L 'tn L d^L  uo% &L r5rceZdZdeffd Ze d deejdee dee dee de e e ff dZ xZS) MLCDVisionTransformerrgct||t|j|jzdz|_t jtjd|j|jzdz|_ y)Nrr8) r"r#rHr$r'vision_rotary_embeddingnn ParameterrRrandn class_pos_embrks r4r#zMLCDVisionTransformer.__init__sh  ':6;M;MQWQkQk;kop;p'q$\\%++a9K9KvOiOi9imn9n*opr5rlrrrrKc||n|jj}||n|jj}||n|jj}| t d|j d|jj z}|j d|jj z}|j||}|j|jj}tj|j|gd}tj||fd}|j|jf} |j|} |j!| } |j#| | |||} | d} | dddddf} |j%| } |s | | f| ddzSt'| | | j(| j*S) Nz You have to specify pixel_valuesrNrrO)rrrrrr8)r pooler_outputrr)rgrrr ValueErrorrnr*rrqrrMrRrtrrry pre_layrnormencoderpost_layernormr rr)r1rlrrrrIrJraembrrencoder_outputsr pooled_outputs r4rbzMLCDVisionTransformer.forwards%9$D $++JjJj &1%r?rr#rrrRrzrrrr rbrCrDs@r4rrsq/q 59,0/3&* / u001/ $D>/ 'tn / d^ / u00 1 / / r5rc&eZdZeZdZdZdZdZdZ y)MLCDPreTrainedModelmlcdTcL|jj}t|tr|jj}tj j |jd|jdz|ztj j |jj|jj|zyt|tr,|jj}|jdzd|jjzdzz|z}|jdz|z}tj j |jj|tj j |jj|tj j |j j|tj j |j"j|yt|t$r|jj}|jj&dzd|jjzdzz|z}d|jj&zdz|z}tj j |j(j|tj j |j*j|yt|t,ry|jj}|jj&|jj.zdzdz|z}tj j |j0d|yt|tj2rJ|j4j6j9|jj6j;dyt|tj<r2|j4%|j4j6j9yyy)zInitialize the weightsr;g)meanstd)rrr<N)rgr- isinstancerfrinitnormal_rs embed_dimrorpr,r|r&rrrrrFr$fc1fc2rr'r LayerNormbiasdatazero_fill_Linear)r1modulefactor in_proj_std out_proj_stdfc_std pos_emb_stds r4 _init_weightsz!MLCDPreTrainedModel._init_weightss// f2 3[[33F GGOOF22&BRBRTXBX[aBaO b GGOOF2299v}}?^?^ag?gO h  .[[33F!++T1q6==;Z;Z7Z_c6cdgmmK",,d2fr?r config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_2_supports_sdparr!r5r4rrs##L&*#!N%r5rcreZdZe ddeej deedeedeedee e ff dZ y) MLCDVisionModelNrlrrrrKc||n|jj}||n|jj}||n|jj}|j ||||S)a Example: ```python >>> import requests >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```)rlrrr)rgrrr vision_model)r1rlrrrs r4rbzMLCDVisionModel.forwardsu>%9$D $++JjJj &1%r?rrrRrzrrrr rbr!r5r4rrst59,0/3&* ( u001( $D>( 'tn ( d^ ( u00 1 ( ( r5r)rrr)2typingrrrrrRtorch.nnrconfiguration_utilsrmodeling_flash_attention_utilsr modeling_outputsr r modeling_utilsr r processing_utilsrutilsrrclip.modeling_cliprrrrrrrllama.modeling_llamarqwen2_vl.modeling_qwen2_vlrr get_loggerr=rrrFrHrfr|rrrrr__all__r!r5r4r s43 3BKF&,;[   H %Y%'Y%x g /D/$?)M?)D/'/dY +Y x6 16 r$%/$%$%N* o* Z r5