UhdZddlZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddlZ ddl mZddlmZmZmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd l m!Z!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,e'jZe.Z/dZ0 dKde jbde2de2de2de2de jbf dZ3dLdZ4dZ5dZ6eGdde$Z7eGdde$Z8eGd d!e$Z9Gd"d#ejtZ;Gd$d%ejtZ< dMd&ejtd'e jbd(e jbd)e jbd*ee jbd+e2d,e2fd-Z=Gd.d/ejtZ>Gd0d1ejtZ?Gd2d3eZ@e%Gd4d5e"ZAGd6d7ejtZBGd8d9ejtZCe%d:;Gd<d=eAZDGd>d?ejtZEGd@dAejtZFe%dB;GdCdDeAZGe%GdEdFeAZHe%dG;GdHdIeAZIgdJZJy)NzPyTorch Siglip model.N) dataclass)AnyCallableOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)_calculate_fan_in_and_fan_out)ACT2FN)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringcan_return_tuplelogging torch_int) SiglipConfigSiglipTextConfigSiglipVisionConfigcd}||d|zz ks ||d|zzkDrtjdd|||z |z }|||z |z }|jd|zdz d|zdz |j|j |t j dz|j||j||y)Ncddtj|tjdz zdz S)N?@)matherfsqrt)xs |/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/siglip/modeling_siglip.pynorm_cdfz _trunc_normal_..norm_cdf,s(dhhq499S>122c99zjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.) stacklevelrr#)minmax) warningswarnuniform_erfinv_mul_r$r&add_clamp_)tensormeanstdabr)lus r(_trunc_normal_r=)s: q1s7{q1s7{ 2  ;  !d(c!"A!d(c!"A OOAEAIq1uqy) NN KKdiin$% KK MMaQMr*r6r7r8r9r:returnctj5t|dd|||j|j |dddy#1swYyxYw)anFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}( ext{mean}, ext{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq ext{mean} \leq b`. NOTE: this 'tf' variant behaves closer to Tensorflow / JAX impl where the bounds [a, b] are applied when sampling the normal distribution with mean=0, std=1.0 and the result is subsequently scaled and shifted by the mean and std args. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value rr"N)torchno_gradr=r3r4)r6r7r8r9r:s r(trunc_normal_tf_rBMsI* $vq#q!, Cd#$$$s 0AAc,t|\}}|dk(r|}n|dk(r|}n |dk(r||zdz }|z }|dk(r$t|tj|dz y|dk(rCt j 5|j tj|dddy|d k(rItjd |z}t j 5|j| |dddytd |#1swYyxYw#1swYyxYw) Nfan_infan_outfan_avgr+truncated_normalg۶%?r8normaluniformrzinvalid distribution ) r rBr$r&r@rAnormal_r1 ValueError) r6scalemode distributionrDrEdenomvariancebounds r(variance_scaling_rSgs3F;OFG x    '!Q&u}H))TYYx%8;N%NO  ! ]]_ 4 NNtyy2N 3 4 4  " !h,' ]]_ + OOUFE * + +0?@@ 4 4 + +s3&C>D >D Dc t|ddy)NrDrGrNrOrSr6s r( lecun_normal_rXsf8:LMr*c t|ddy)NrDrIrUrVrWs r(default_flax_embed_initrZsf8(Cr*ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y)SiglipVisionModelOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r]rr@ FloatTensor__annotations__r^r_rr`r*r(r\r\sr*15L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r*r\ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y)SiglipTextModelOutputa Base class for text model's outputs that also contains a pooling of the last hidden states. Args: text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N text_embedsr^.r_r`) rarbrcrdrjrr@rerfr^r_rr`rgr*r(ririsr*04K%++,359x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r*riceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd Zy) SiglipOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`SiglipTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`SiglipVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`SiglipTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`SiglipVisionModel`]. Nlosslogits_per_imagelogits_per_textrjr]text_model_outputvision_model_outputr>cHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))rprqN)getattrto_tuple).0kselfs r( z(SiglipOutput.to_tuple..s=  LLDGRYZ^`aRbRkRkRm m s-0)tuplekeysrxs`r(ruzSiglipOutput.to_tuples# YY[   r*)rarbrcrdrmrr@rerfrnrorjr]rprrqrrrurgr*r(rlrls()-D(5$$ %,48hu001837OXe//07/3K%++,304L(5,,-448186:3: %* r*rlceZdZdeffd Zdej dededej fdZd dejdej fd Z xZ S) SiglipVisionEmbeddingsconfigcft|||_|j|_|j |_|j |_tj|j|j|j |j d|_ |j |j zdz|_ |j|_ tj|j|j|_|jdt!j"|jj%ddy)Nvalid) in_channels out_channels kernel_sizestridepaddingr+ position_idsrF persistent)super__init__r hidden_size embed_dim image_size patch_sizer Conv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr@arangeexpandrxr __class__s r(rzSiglipVisionEmbeddings.__init__s  ++ ++ ++!yy++??  !OOt>1D!--"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr* embeddingsheightwidthr>c|jd}|jjjd}tjj s%||k(r ||k(r|j|j S|jjjd}|jd}||jz}||jz} t|dz} |jd| | |}|jdddd}tjj||| fdd }|jddddjdd|}|S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and no class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrrg?rr+bicubicF)sizerN align_corners)shaperweightr@jit is_tracingr unsqueezerrreshapepermuter functional interpolateview) rxrrrrrpatch_pos_embeddim new_height new_widthsqrt_num_positionss r(interpolate_pos_encodingz/SiglipVisionEmbeddings.interpolate_pos_encodingsE!&&q) //66<O>O&PPJr*F) rarbrcrrr@Tensorintrrer __classcell__rs@r(r~r~s`q1q($5<<$$UX$]b]i]i$L E$5$5 Z_ZfZf r*r~c eZdZdeffd Z ddeejdeejdeejdejfdZ xZ S) SiglipTextEmbeddingsrcNt||j}tj|j ||_tj|j||_|jdtj|jjddy)NrrFr) rrrr r vocab_sizetoken_embeddingmax_position_embeddingsrrr@rrrxrrrs r(rzSiglipTextEmbeddings.__init__4s && !||F,=,=yI"$,,v/M/My"Y  ELL)G)GHOOPWXej  r* input_idsr inputs_embedsr>c8||jdn|jd}|jjjd}||kDrtd|d|||jddd|f}||j |}|j|}||z}|S)NrrzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rrrrLrr)rxrrr seq_lengthmax_position_embeddingposition_embeddingsrs r(rzSiglipTextEmbeddings.forward@s -6,AY__R(}GZGZ[]G^ !%!8!8!?!?!E!Ea!H . .d,<=S> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r*r_routput_attentionsr>c |j\}}}|j|}|j|}|j|} |j |||j |j jdd}|j |||j |j jdd}| j |||j |j jdd} t} |jjdk7rN|jjdk(r|rtjdnt|jj} | |||| ||j|j|j sdn |j"\} } | j%|||j'} |j)| } |sd} | | fS) z#Input shape: Batch x Time x Channelrr+eagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrrN)rrrrrrrrrr_attn_implementationlogger warning_oncerrrMrrrrr) rxr_rr batch_sizerrqueriesr{valuesattention_interfacerrs r(rzSiglipAttention.forwards-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc(? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0  LL((r*NF)rarbrcrdrrrrr@rrboolrrrrs@r(rrrs{GBu%79I%IJB.26,1 -)||-)!.-)$D> -) u||Xell33 4 -)r*rcVeZdZfdZdej dej fdZxZS) SiglipMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) rrrr hidden_act activation_fnr rrintermediate_sizefc1fc2rs r(rzSiglipMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr*r_r>cl|j|}|j|}|j|}|Sr)r rr )rxr_s r(rzSiglipMLP.forwards4/ **=9 / r*)rarbrcrr@rrrrs@r(rrs$KU\\ellr*rc eZdZdeeefffd Z ddejdejde e de ejfdZ xZS) SiglipEncoderLayerrcDt||j|_t j |j|j |_t||_ t j |j|j |_ t||_ y)Neps) rrrrr LayerNormlayer_norm_eps layer_norm1r self_attn layer_norm2rmlprs r(rzSiglipEncoderLayer.__init__sm ++<<Fc|}|j|}|j|||\}}||z}|}|j|}|j|}||z}|f}|r||fz }|S)a= Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r_rr)rrrr)rxr_rrresidualroutputss r(rzSiglipEncoderLayer.forwards !((7 &*nn')/'5' # | !=0  ((7 /  =0  "    &Gr*r)rarbrcrrrrr@rrrrrerrrs@r(r r sb%u%79I%IJ%-2 $||$ $$D> $ u  ! $r*r c.eZdZeZdZdZgdZdZdZ dZ y)SiglipPreTrainedModelsiglipT)rr r~r #SiglipMultiheadAttentionPoolingHeadc t|trt|jtr |jjj n|jj }t jj|jjdtj|z yt|t jrt|jyt|trt jj!|j"jt jj!|j$jt jj!|j&jt jj!|j(jt jj+|j"j,t jj+|j$j,t jj+|j&j,t jj+|j(j,yt|t.rt jj!|j0jt jj!|j2jt jj|j0j,dt jj|j2j,dyt|t4rt jj!|j6j8t jj!|j:j<j8t jj+|j:j>j8yt|t@rrtCjDtCjFd}|jHj8jK||jLj8jOyt|tPrnt jj|jRj|jjj dz|jjTzyt|t jVt jXfrLt[|j|j,*t jj+|j,yyt|t j\rJ|j,j8jO|jj8jKdyy)zInitialize the weightsrrHgư>r"rN)/ isinstancer~rr vision_configrr initrKrrnpr&rrZrxavier_uniform_rrrrzeros_biasrr r rprobedata attentionin_proj_weight in_proj_bias SiglipModelr@logr6 logit_scalefill_ logit_biaszero_SiglipForImageClassification classifierinitializer_factorrrrXr)rxrrlogit_scale_inits r( _init_weightsz#SiglipPreTrainedModel._init_weights s f4 5dkk<8 ))55[[,,  GGOOF55<  ,$yyc):;     # # ) )*: ;    " " ( ( *  < = GGOO!!((KK--994?$++B`B``  BII 6 7 &-- ({{&v{{+'  - KK   " " $ MM   $ $S ).r*N) rarbrcr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_2_supports_sdpar5rgr*r(rrs-L &*#"N,*r*rc xeZdZdZdeffd Ze d deejdee dee de fdZ xZ S) SiglipEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`SiglipEncoderLayer`]. Args: config: SiglipConfig rct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rrrr ModuleListrangenum_hidden_layersr layersgradient_checkpointing)rxrrrs r(rzSiglipEncoder.__init__BsO  mmvOgOgIh$iA%7%?$ij &+#%jsA#rroutput_hidden_statesr>c||n|jj}||n|jj}|rdnd}|rdnd}|}|jD]&}|r||fz}||||} | d}|s|| dfz}(|r||fz}t |||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nrg)rrr)r^r_r`)rrrDrBr) rxrrrrDencoder_statesall_attentionsr_ encoder_layer layer_outputss r(rzSiglipEncoder.forwardIs<2C1N-TXT_T_TqTq$8$D $++JjJj  40d% ![[ FM#!/=2B!B)"3M *!,M !/=3C2E!E F +}.>>N+(%  r*r)rarbrcrdrrrrr@rrrrrrs@r(r=r=9sl,|,26,0/3 < !.< $D> < 'tn <  < < r*r=ceZdZdeffd Zee d deejdeejdeejdee dee de f d Z xZ S) SiglipTextTransformerrcFt|||_|j}t ||_t ||_tj||j|_ tj||j|_|jdk(|_y)Nrflash_attention_2)rrrrrrr=encoderr rrfinal_layer_normrprojection_sizeheadr_use_flash_attention_2rs r(rzSiglipTextTransformer.__init__s}  && .v6$V, " YFc ||n|jj}||n|jj}| td|j }|j d|d}|j ||}|"|jst||j}|j||||}|j} |j| } | dddddf} |j| } t| | |j|j S)NzYou have to specify input_idsr)rr)rrrrDr^ pooler_outputr_r`)rrrDrLrrrrRrrrNr^rOrQrr_r`) rxrrrrrD input_shaper_encoder_outputsr^ pooled_outputs r(rzSiglipTextTransformer.forwards&2C1N-TXT_T_TqTq$8$D $++JjJj   <= =nn& NN2{27 ),W   %d.I.I7 H[H[\N+/<<')/!5 ,8, ,== 112CD*!R(3  -0 )/')77&11   r*NNNNN)rarbrcrrrrrr@rrrrrrs@r(rKrKs Y/ Y-115/3,0/3 . ELL). !.. u||, . $D> . 'tn .  $. . r*rKzK The text model from SigLIP without any head or projection on top. ) custom_introceZdZeZdeffd ZdejfdZdZ e e d de e jde e jde e jd e ed e edef d ZxZS) SiglipTextModelrcdt||t||_|j yr)rrrK text_model post_initrs r(rzSiglipTextModel.__init__s&  /7 r*r>cB|jjjSrr^rrr|s r(get_input_embeddingsz$SiglipTextModel.get_input_embeddingss))999r*c:||jj_yrra)rxrs r(set_input_embeddingsz$SiglipTextModel.set_input_embeddingss5:""2r*rrrrrDc.|j|||||S)a Examples: ```python >>> from transformers import AutoTokenizer, SiglipTextModel >>> model = SiglipTextModel.from_pretrained("google/siglip-base-patch16-224") >>> tokenizer = AutoTokenizer.from_pretrained("google/siglip-base-patch16-224") >>> # important: make sure to set padding="max_length" as that's how the model was trained >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding="max_length", return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrrrD)r^)rxrrrrrDs r(rzSiglipTextModel.forwards)6)%/!5   r*rY)rarbrcrr6rr Modulerbrdrrrr@rrrrrrs@r(r\r\s $L/ :bii:;-115/3,0/3  ELL) !. u||,  $D>  'tn   $  r*r\c jeZdZdeffd Zee ddeedeedeede fdZ xZ S) SiglipVisionTransformerrcLt|||_|j}t ||_t ||_tj||j|_ t|dsdn |j|_|jrt||_yy)Nrvision_use_headT)rrrrr~rr=rNr rrpost_layernormhasattrrkuse_headrrQrs r(rz SiglipVisionTransformer.__init__s  && 08$V,  ll9&:O:OP$+F4E$FFLbLb ==;FCDI r*rrDrr>cv||n|jj}||n|jj}|j||}|j |||}|j }|j |}|jr|j|nd}t|||j|jS)N)r)rrrDrT) rrrDrrNr^rlrnrQrr_r`) rxrrrDrr_rWr^rUs r(rzSiglipVisionTransformer.forward s2C1N-TXT_T_TqTq$8$D $++JjJj  Ogh +/<<'/!5,8,  ,== //0AB8<  "344 )/')77&11   r*NNF) rarbrcrrrrrrrrrrs@r(rirism D1 D-1/338  $D> 'tn  #+4.  $   r*ric.eZdZdZdeffd ZdZxZS)rzMultihead Attention Pooling.rct|tjt j dd|j |_tjj|j |jd|_ tj|j |j|_ t||_y)NrT) batch_firstr)rrr Parameterr@randnrr&MultiheadAttentionrr(rr layernormrrrs r(rz,SiglipMultiheadAttentionPoolingHead.__init__3s \\%++aF4F4F"GH 44V5G5GIcIcqu4vf&8&8f>S>STV$r*c|jd}|jj|dd}|j|||d}|}|j |}||j |z}|dddfS)Nrr)rr&repeatr(rwr)rx hidden_staterr&rs r(rz+SiglipMultiheadAttentionPoolingHead.forward;sv!''*  !!*a3~~e\<HK ~~l3 $((<"88 AqD!!r*)rarbrcrdrrrrrs@r(rr0s&%1% "r*rzM The vision model from SigLIP without any head or projection on top. c eZdZeZdZdeffd ZdejfdZ e e d de e de e de defd ZxZS) SiglipVisionModelrrcdt||t||_|j yr)rrri vision_modelr_rs r(rzSiglipVisionModel.__init__Qs)  3F; r*r>cB|jjjSr)r~rrr|s r(rbz&SiglipVisionModel.get_input_embeddingsYs  ++;;;r*rrDrc,|j||||S)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, SiglipVisionModel >>> model = SiglipVisionModel.from_pretrained("google/siglip-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled features ```rrrDr)r~)rxrrrDrs r(rzSiglipVisionModel.forward\s(<  %/!5%= !  r*rp)rarbrcrr6main_input_namerr rgrbrrrrrrrrs@r(r|r|Hs &L$O1! 'tn ! #' ! $ ! ! r*r|ceZdZeZdeffd Ze ddeejdeejdeejdee dee dejf d Z e dd eejdee dee d e dejf d Z ee ddeejd eejdeejdeejd ee dee dee d e defdZxZS)r+rct||t|jts"t dt |jdt|jts"t dt |jd|j}|j}tj|}tj|}|j|_ |j|_ tjt!j"d|_tjt!j"d|_|j)y)NzMconfig.text_config is expected to be of type SiglipTextConfig but is of type .zQconfig.vision_config is expected to be of type SiglipVisionConfig but is of type r)rrr text_configr TypeErrortyper rr\ _from_configr|r^r~r rtr@rur-r/r_)rxrrr r^r~rs r(rzSiglipModel.__init__s  &,,.>?++,-Q0  &..0BC--./q2  (( ,, %11+> (55mD %//(55<< A7,,u{{1~6 r*rrrrrDr>c||n|jj}||n|jj}|j|||||}|j}|S)aJ Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`SiglipTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip-base-patch16-224") >>> tokenizer = AutoTokenizer.from_pretrained("google/siglip-base-patch16-224") >>> # important: make sure to set padding="max_length" as that's how the model was trained >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding="max_length", return_tensors="pt") >>> with torch.no_grad(): ... text_features = model.get_text_features(**inputs) ```rf)rrrDr^rU)rxrrrrrD text_outputsrXs r(get_text_featureszSiglipModel.get_text_featuressr:2C1N-TXT_T_TqTq$8$D $++JjJj 48??)%/!5 4C4 %22 r*rrc||n|jj}||n|jj}|j||||}|j}|S)a Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`SiglipVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... image_features = model.get_image_features(**inputs) ```r)rrrDr~rU)rxrrrDrvision_outputsrXs r(get_image_featureszSiglipModel.get_image_featuressrB2C1N-TXT_T_TqTq$8$D $++JjJj 6:5F5F%/!5%= 6G6 '44 r* return_lossc ||n|jj}||n|jj}|j||||} |j |||||} | j } | j } | | j dddz } | | j dddz } tj| | jj| j} |jj| j|jj| j}}| |jz|z} | j}d}|rtj| j!d| j }tj"|  d|zz}tj$j&j)|| z}tj*|d  }|j-}t/||| | | | | S) a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a photo of 2 cats", "a photo of 2 dogs"] >>> # important: we pass `padding=max_length` since the model was trained with this >>> inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image >>> probs = torch.sigmoid(logits_per_image) # these are the probabilities >>> print(f"{probs[0][0]:.1%} that image 0 is '{texts[0]}'") 31.9% that image 0 is 'a photo of 2 cats' ```Nrrfr+rT)rrkeepdimr)devicer)rmrnrorjr]rprq)rrrDr~r^rUnormr@rtrrr-r/expeyer ones_liker r logsigmoidsumr7rl)rxrrrrrrrDrrrr]rjror-r/rnrmrm1_diag1logliknlls r(rzSiglipModel.forwardsX2C1N-TXT_T_TqTq$8$D $++JjJj 6:5F5F%/!5%= 6G6 48??)%/!5 4C4 &33 "00 $l&7&7!T&7&RR !K$4$4qb$$4$OO  ,,{LNN4D4G4G HZHZ4[\"&"2"2"5"5k6H6H"I4??K]K]^i^p^pKqZ )KOO,== J*,,. ))O003O + 'tn +   ++Z59,0/3). .u001.$D>.'tn . #' .    ..`15481537&*,0/3).^ E,,-^ u001^ !. ^ u//0 ^ d^ ^ $D>^ 'tn^ #'^  ^ ^ r*r+z SigLIP vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of the patch tokens) e.g. for ImageNet. ceZdZdZdeddffd Zee d dee jdee jdee dee d e de f d Z xZS) r1rrr>Nc~t|||j|_tj |j }|j |_|jdkDr4tj|j j|jntj|_ |jy)Nr) rr num_labelsr|rr r~r rrIdentityr2r_)rxrr~rs r(rz%SiglipForImageClassification.__init__qs   ++)55f6J6JK (55OUN_N_bcNcBIIf**668I8I Jikititiv  r*labelsrrDrc||n|jj}||n|jj}|j||||}|j}t j |d}|j|}d} ||j|j}|jj|jdk(rd|j_ nl|jdkDrL|jt jk(s|jt jk(rd|j_ nd|j_ |jjdk(rIt} |jdk(r&| |j!|j!} n| ||} n|jjdk(r=t#} | |j%d|j|j%d} n,|jjdk(rt'} | ||} t)| ||j*|j, S) a$ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, SiglipForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a `SiglipModel` from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random if seed is not set above. >>> image_processor = AutoImageProcessor.from_pretrained("google/siglip-base-patch16-224") >>> model = SiglipForImageClassification.from_pretrained("google/siglip-base-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the two classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: LABEL_1 ```N)rrDrrr regressionsingle_label_classificationmulti_label_classificationr)rmlogitsr_r`)rrrDr~r^r@r7r2rr problem_typerrlongrr squeezer rr rr_r`) rxrrrrDrrsequence_outputrrmloss_fcts r(rz$SiglipForImageClassification.forwardsR2C1N-TXT_T_TqTq$8$D $++JjJj /3.?.? /!5%= /@/ "33 **_!<1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./$!//))   r*)NNNNF)rarbrcrrrrrrr@rrrrrrs@r(r1r1hs%O|$04)-,0/3). X u||,X &X $D> X 'tn X #' X  X X r*r1)r+rr\r|r1)rr"gr#)r"rDrI)r)Krdr$r/ dataclassesrtypingrrrrrnumpyr"r@torch.utils.checkpointr torch.nnr r r torch.nn.initr activationsrmodeling_attn_mask_utilsrmodeling_layersrmodeling_outputsrrrmodeling_utilsrrutilsrrrrrconfiguration_sigliprrr get_loggerrarr=rfloatrBrSrXrZr\rirlrgr~rrrrr rr=rKr\rirr|r+r1__all__rgr*r(rs !88 AA7!B9bbFVVTT   H %! J\_$ LL$ %$27$BG$SX$ \\$4A2ND ?k? ?8 ?K? ?8 ! ;!  ! HERYYER%299%^% II% <<% % << % U\\* %  %%.D)biiD)P  -3-`;*O;*;*~M BIIM `< BII< ~ 0 +0  0 f- bii- `""))"0 2 -2  2 jb 'b b J o #8o  o d r*