UhW6,dZddlmZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddl m Z mZddlmZdd lmZmZdd lmZmZdd lmZdd lmZmZmZmZmZd dlm Z m!Z!m"Z"erddl#m$Z$ejJe&Z'de jde jfdZ(de jde jfdZ)eGddeZ*de de fdZ+de de fdZ,dZ-dZ.eGddeZ/eGdd eZ0Gd!d"ejbZ2Gd#d$ejbZ3Gd%d&ejbZ4Gd'd(ejbZ5Gd)d*ejbZ6eGd+d,eZ7Gd-d.ejbZ8Gd/d0ejbZ9Gd1d2e7Z:Gd3d4ejbZ;Gd5d6e7ZGd;de7Z@gd?ZAy)@zPyTorch OWLv2 model.) dataclass) lru_cache)AnyDictOptionalTupleUnionN)Tensornn)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstringis_vision_availablelogging torch_int) Owlv2ConfigOwlv2TextConfigOwlv2VisionConfig)center_to_corners_formatlogitsreturnctjj|tjt ||j S)Ndevice)r functional cross_entropytorcharangelenr!)rs z/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/owlv2/modeling_owlv2.pycontrastive_lossr(,s/ == & &vu||CKPVP]P]/^ __ similaritycZt|}t|j}||zdz S)Ng@)r(t)r* caption_loss image_losss r' owlv2_lossr/1s,#J/L!*,,.1J : % ,,r)ceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd Zy) Owlv2Outputa! Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size * num_max_text_queries, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Owlv2TextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Owlv2VisionModel`]. text_model_output (Tuple[`BaseModelOutputWithPooling`]): The output of the [`Owlv2TextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`Owlv2VisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrcHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw)r7r8Ngetattrto_tuple.0kselfs r' z'Owlv2Output.to_tuple..W=  LLDGRYZ^`aRbRkRkRm m -0tuplekeysrBs`r'r>zOwlv2Output.to_tupleV# YY[   r))__name__ __module__ __qualname____doc__r2rr$ FloatTensor__annotations__r3r4r5r6r7rr8rrr>r)r'r1r17s*)-D(5$$ %,48hu001837OXe//07/3K%++,304L(5,,-448186:3: %* r)r1r,c|jr>|jtjtjfvr|S|j S|jtj tjfvr|S|jSN) is_floating_pointdtyper$float32float64floatint32int64int)r,s r'_upcastr\^s`GG u}}==qL1779LGG U[[99qFquuwFr)boxescft|}|dddf|dddfz |dddf|dddfz zS)a Computes the area of a set of bounding boxes, which are specified by its (x1, y1, x2, y2) coordinates. Args: boxes (`torch.FloatTensor` of shape `(number_of_boxes, 4)`): Boxes for which the area will be computed. They are expected to be in (x1, y1, x2, y2) format with `0 <= x1 < x2` and `0 <= y1 < y2`. Returns: `torch.FloatTensor`: a tensor containing the area for each box. Nrr r)r\)r]s r'box_arear`gsB ENE !Q$K%1+ %%1+ad *C DDr)c^t|}t|}tj|dddddf|ddddf}tj|dddddf|ddddf}||z j d}|dddddf|dddddfz}|dddf|z|z }||z } | |fS)Nr_rminr)r`r$maxrcclamp) boxes1boxes2area1area2left_top right_bottom width_heightinterunionious r'box_iourpxs V E V Eyy4! ,fQUm.rDrErFrIs`r'r>z#Owlv2ObjectDetectionOutput.to_tuplerJr))rKrLrMrNr2rr$rOrPrzrrr{r|r5r6r}r7rr8rrr>rQr)r'ryrys D)-D(5$$ %, $Ix~$*.FHU&& '.59x 1 129.2J**+2/3K%++,304L(5,,-404L(5,,-448186:3: %* r)ryc0eZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZeed <dZeed <d eefd Zy) %Owlv2ImageGuidedObjectDetectionOutputa Output type of [`Owlv2ForObjectDetection.image_guided_detection`]. Args: logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`): Classification logits (including no-object) for all queries. target_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual target image in the batch (disregarding possible padding). You can use [`~Owlv2ImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. query_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual query image in the batch (disregarding possible padding). You can use [`~Owlv2ImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`Owlv2VisionModel`]. OWLv2 represents images as a set of image patches and computes image embeddings for each patch. query_image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`Owlv2VisionModel`]. OWLv2 represents images as a set of image patches and computes image embeddings for each patch. class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Class embeddings of all image patches. OWLv2 represents images as a set of image patches where the total number of patches is (image_size / patch_size)**2. text_model_output (Tuple[`BaseModelOutputWithPooling`]): The output of the [`Owlv2TextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`Owlv2VisionModel`]. Nrr6query_image_embedstarget_pred_boxesquery_pred_boxesr}r7r8rcHtfdjDS)Nc3dK|]'}|dvr|nt|j)ywr;r<r?s r'rCzAOwlv2ImageGuidedObjectDetectionOutput.to_tuple..rDrErFrIs`r'r>z.Owlv2ImageGuidedObjectDetectionOutput.to_tuplerJr))rKrLrMrNrrr$rOrPr6rrrr}r7rr8rrr>rQr)r'rrs>+/FHU&& '.04L(5,,-46:!2!23:59x 1 12948hu001804L(5,,-448186:3: %* r)rceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) Owlv2VisionEmbeddingsconfigct||j|_||_|j|_t jtj|j|_ t j|j|j |j|jd|_ |j|jzdz|_|jdz|_t j"|j |j |_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestridebiasr_r position_idsr persistent)super__init__ patch_sizer hidden_size embed_dimr Parameterr$randnclass_embeddingConv2d num_channelspatch_embedding image_size num_patches num_positions Embeddingposition_embeddingregister_bufferr%expandrBr __class__s r'rzOwlv2VisionEmbeddings.__init__ s  ++ ++!||EKK8J8J,KL!yy++))$$  #--1B1BBqH!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr) embeddingsheightwidthrc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrg?r r_bicubicF)sizemode align_cornersdim)shaperweight unsqueezer$jit is_tracingrrrreshapepermuter r" interpolateviewcat) rBrrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r'interpolate_pos_encodingz.Owlv2VisionEmbeddings.interpolate_pos_encoding sv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr) pixel_valuesrch|j\}}}}|j|}|jdjdd}|jj |dd}t j||gd} |r| |j| ||z} | S| |j|jz} | S)Nr_rrr) rrflatten transposerrr$rrrr) rBrr batch_size_rr patch_embedsr}rs r'forwardzOwlv2VisionEmbeddings.forwardIs'3'9'9$ Avu++L9 #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr)F) rKrLrMrrr$r r[rrOboolr __classcell__rs@r'rr smq0q*'D5<<'D'DUX'D]b]i]i'DR E$5$5 QU bgbnbn r)rc eZdZdeffd Z ddeejdeejdeejdejfdZ xZ S) Owlv2TextEmbeddingsrc^t|tj|j|j |_tj|j|j |_|jdtj|jjddy)NrrFr) rrr r vocab_sizertoken_embeddingmax_position_embeddingsrrr$r%rrs r'rzOwlv2TextEmbeddings.__init__Ys !||F,=,=v?Q?QR"$,,v/M/MvOaOa"b  ELL)G)GHOOPWXej  r) input_idsr inputs_embedsrc||jdn|jd}||jddd|f}||j|}|j|}||z}|S)Nr)rrrr)rBrrr seq_lengthposition_embeddingsrs r'rzOwlv2TextEmbeddings.forwardcs{ -6,AY__R(}GZGZ[]G^  ,,Q  ^> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r)tensorseq_lenbszc|j|||j|jjddj S)Nrr_)rrrr contiguous)rBrrrs r'_shapezOwlv2Attention._shapes7{{3GQQRSUVWbbddr) hidden_statesattention_maskcausal_attention_maskoutput_attentionsrc|j\}}}|j||jz}|j|j |d|} |j|j |d|} ||j zd|jf} |j|||j| }| j| } | j| } | jd} tj|| jdd} | j||j z|| fk7r/td||j z|| fd| j|{|j|d|| fk7r#td|d|| fd|j| j||j || |z} | j||j z|| } |{|j|d|| fk7r#td|d|| fd|j| j||j || |z} | j||j z|| } tjj| d} |r?| j||j || }|j||j z|| } nd}tjj!| |j |j" }|j%| j&}tj|| }|j||j z||jfk7r7td ||j ||jfd|j|j||j ||j}|jdd}|j)|||}|j+|}||fS) z#Input shape: Batch x Time x Channelrrr_z$Attention weights should be of size z , but is Nz!Attention mask should be of size r)ptrainingz `attn_output` should be of size )rrrrrrrrrr$bmmrrsr r"softmaxrrtorUrr)rBrrrrrtgt_lenr query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r'rzOwlv2Attention.forwards#0"4"4"6Wi{{=1DJJ> [[]!;RE {{4;;}#=r3G DNN*B > Ct{{<#>CCZP $Z__j1 (|((*5 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*  ! ,$))+Q/II 7a'8R7ST-22457(,,S$..'7SVkkL',,S4>>-A7GTL  %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4< O2 u||Xell3XeELL>Q5RR S O2r)rcVeZdZfdZdej dej fdZxZS)Owlv2MLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yrS) rrrr hidden_act activation_fnr rrintermediate_sizefc1fc2rs r'rzOwlv2MLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr)rrcl|j|}|j|}|j|}|SrS)r r r )rBrs r'rzOwlv2MLP.forwards4/ **=9 / r))rKrLrMrr$r rrrs@r'rrs$KU\\ellr)rc eZdZdeffd Z d dej dej dej deede ejf dZ xZ S) Owlv2EncoderLayerrcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yNeps) rrrrr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rs r'rzOwlv2EncoderLayer.__init__sm ++'/<<F & u  ! &r)rc$eZdZeZdZdZdgZdZy)Owlv2PreTrainedModelowlv2TrcL |jj}t|trj|jj j jd|dz|jj j jd|dznt|tr|jj}tjj|jd|jdz|ztjj|jj |jj|ztjj|jj |jj|znt|t r-|jj}|jdzd|jj"zdzz|z}|jdz|z}tjj|j$j |tjj|j&j |tjj|j(j |tjj|j*j |nt|t,r|jj}|jj.dzd|jj"zdzz|z}d|jj.zdz|z}tjj|j0j |tjj|j2j |nt|t4rtjj|j6j |j8dz|jjztjj|j:j |j<dz|jjzt|tj>rI|j@j jC|j j jEdt|tjFr2|j@%|j@j jCyyy) zInitialize the weightsg{Gz?)meanstdr)r$r_?N)$rinitializer_factor isinstancerrrdatanormal_rrr initrrrinitializer_rangernum_hidden_layersrrrrrrr r  Owlv2Modeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrrzero_fill_r)rBmodulefactor in_proj_std out_proj_stdfc_stds r' _init_weightsz"Owlv2PreTrainedModel._init_weights.s// f1 2  " " ) ) . . 6 6CVd] 6 S  % % , , 1 1 9 9sQU 9 V  5 6[[33F GGOOF22&BRBRTXBX[aBaO b GGOOF2299v}}?^?^ag?gO h GGOOF55<<&--BaBadjBjO k  /[[33F!++T1q6==;Z;Z7Z_c6cdgmmK",,d2f$hi &+#%isArrroutput_hidden_states return_dictrc||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|} |jD]_} |r|| fz}|j r,|j r |j| j| |||} n | | |||} | d} |sW|| dfz}a|r|| fz}|std| ||fDSt| ||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`). attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrQ)rrrc3&K|] }|| ywrSrQ)r@vs r'rCz'Owlv2Encoder.forward..seqWXWde)last_hidden_stater attentions) rrrEuse_return_dictrCrDr_gradient_checkpointing_func__call__rGr) rBrrrrrErFencoder_statesall_attentionsr encoder_layer layer_outputss r'rzOwlv2Encoder.forwardfs=>2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r)NNNNN)rKrLrMrNrrrr$r rr rrrrrs@r'r?r?Xs,{,268<,0/3&*H !.H  ( 5 H $D> H 'tn H d^H  uo% &H r)r?ceZdZdeffd Ze d dejdeejdeejdee dee dee d e e e ffd Z xZS) Owlv2TextTransformerrct|||_|j}t ||_t ||_tj||j|_ yr) rrrrrrr?encoderr rrfinal_layer_norm)rBrrrs r'rzOwlv2TextTransformer.__init__sQ  && -f5#F+ " YF ? 'tn ? d^?  u00 1? ? r)rVceZdZeZdeffd ZdejfdZdZ e d de jde e jde ed e ed e edeeeff d ZxZS) Owlv2TextModelrcdt||t||_|j yrS)rrrV text_model post_initrs r'rzOwlv2TextModel.__init__s&  .v6 r)rcB|jjjSrSrdrrrIs r'get_input_embeddingsz#Owlv2TextModel.get_input_embeddingss))999r)c:||jj_yrSrg)rBvalues r'set_input_embeddingsz#Owlv2TextModel.set_input_embeddings s5:""2r)rrrrErFc.|j|||||S)a input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import AutoProcessor, Owlv2TextModel >>> model = Owlv2TextModel.from_pretrained("google/owlv2-base-patch16") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrrErF)rd)rBrrrrErFs r'rzOwlv2TextModel.forwards)<)/!5#   r))NNNN)rKrLrMrr:rr Modulerhrkrr$r rrr rrrrrs@r'rbrbs"L :bii:;26,0/3&* # <<# !.# $D> # 'tn # d^ #  u00 1# # r)rbceZdZdeffd Ze d dejdee dee dee dee de e e ff d Z xZS) Owlv2VisionTransformerrc0t|||_t||_t j |j|j|_ t||_ t j |j|j|_ yr) rrrrrr rrr pre_layernormr?rXpost_layernormrs r'rzOwlv2VisionTransformer.__init__7sk  /7\\&*<*<&BWBWX#F+  ll6+=+=6CXCXYr)rrrErrFrc(||n|jj}||n|jj}||n|jj}|jj j j}|j|}|j ||}|j|}|j||||}|d} | dddddf} |j| } |s | | f|ddzSt| | |j|jS)N)r)rrrErFrrr[)rrrErMrrrrUrrrrXrsrrrL) rBrrrErrFexpected_input_dtyperr_rKr`s r'rzOwlv2VisionTransformer.forward@s-2C1N-TXT_T_TqTq$8$D $++JjJj &1%>EEKK#';<  Ogh **=9 ,,'/!5# ' ,A.)!Q'2 ++M: %}58KK K)/')77&11   r))NNFN)rKrLrMrrrr$rOrrr rrrrrs@r'rprp6sZ0Z-1/338&* ) '') $D>) 'tn ) #+4. ) d^ )  u00 1) ) r)rpceZdZeZdZdeffd ZdejfdZ e d de e jde ede eded e edeeeff d ZxZS) Owlv2VisionModelrrcdt||t||_|j yrS)rrrp vision_modelrers r'rzOwlv2VisionModel.__init__rs'  26: r)rcB|jjjSrS)ryrrrIs r'rhz%Owlv2VisionModel.get_input_embeddingsxs  ++;;;r)rrErrFc.|j|||||S)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Owlv2VisionModel >>> model = Owlv2VisionModel.from_pretrained("google/owlv2-base-patch16") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrErrF)ry)rBrrrErrFs r'rzOwlv2VisionModel.forward{s+6  %/!5%=# !  r)NNNFN)rKrLrMrr:main_input_namerr rnrhrrr$rOrr rrrrrs@r'rwrwns$L$O0   'tn  #'  d^   u00 1    r)rwceZdZeZdeffd Ze ddeejdeejdee dee dee dejf d Z e dd eejdee dee d e dee dejf d Z e ddeejd eejdeejd ee dee dee d e dee dee deeeffdZxZS)r-rc<t||t|jts"t dt |jdt|jts"t dt |jd|j}|j}|j|_ |j|_ |j|_ t||_t||_t#j$|j|jd|_t#j$|j|jd|_t#j*t-j.|j0|_|j5y)NzLconfig.text_config is expected to be of type Owlv2TextConfig but is of type .zPconfig.vision_config is expected to be of type Owlv2VisionConfig but is of type F)r)rrr' text_configr TypeErrortype vision_configrprojection_dimrr/r1rVrdrpryr rr0r.rr$rlogit_scale_init_value logit_scalere)rBrrrrs r'rzOwlv2Model.__init__sR  &,,o>++,-Q0  &..0AB--./q2  (( ,, $33)55 - 9 9.{;2=A!#4+@+@$BUBU\a!b!yy)<)Q>QX]^<< V5R5R(ST r)rrrrErFrc||n|jj}|j|||}|d}|j|}|S)aL input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Owlv2TextModel`]. Examples: ```python >>> from transformers import AutoProcessor, Owlv2Model >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> text_features = model.get_text_features(**inputs) ```)rrrFr)rrMrdr.) rBrrrrErF text_outputr` text_featuress r'get_text_featureszOwlv2Model.get_text_featuressP>&1%>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Owlv2Model >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```r|r)rrrErMryr0) rBrrrErrFvision_outputsr`image_featuress r'get_image_featureszOwlv2Model.get_image_featuress82C1N-TXT_T_TqTq$8$D $++JjJj &1%r) return_lossreturn_base_image_embedsc (||n|jj}||n|jj}| | n|jj} |j ||||| } |j ||||| } | d} |j | } | d} |j| } | tjj| dddz } | tjj| dddz }|jjj| j}tj|| j!|z}|j!}d}|r t#|}|} | s||| | | | f}||f|zS|St%|||| | | | S) a4 return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. return_base_image_embeds (`bool`, *optional*): Whether or not to return the base image embeddings. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Owlv2Model >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=[["a photo of a cat", "a photo of a dog"]], images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nr|rmrr_rT)ordrkeepdim)r2r3r4r5r6r7r8)rrrErMryrdr.r0r$linalgnormrexprr!matmulr,r/r1)rBrrrrrrErrrFr text_outputsr5r6text_embeds_normrr4r3r2outputs r'rzOwlv2Model.forwardsF2C1N-TXT_T_TqTq$8$D $++JjJj &1% % 'tn % d^ %   %%N59,0/3).&* ,u001,$D>,'tn , #' , d^ ,   ,,\154815&*,0/3).37&*Z E,,-Z u001Z !. Z d^ Z $D> Z 'tnZ #'Z #+4.Z d^Z  uk! "Z Z r)r-cbeZdZddedeffd ZdejdejfdZ xZ S)Owlv2BoxPredictionHeadrout_dimc"t||jj}t j |||_t j |||_t j|_ t j |||_ yrS) rrrrr rdense0dense1GELUgeludense2)rBrrrrs r'rzOwlv2BoxPredictionHead.__init__{sb $$00iiu- iiu- GGI iiw/ r)rrc|j|}|j|}|j|}|j|}|j|}|SrS)rrrr)rBrrs r'rzOwlv2BoxPredictionHead.forwardsM^,6"V$6"V$ r))) rKrLrMrr[rr$r rOrrrs@r'rrzs30{0S0ellu7H7Hr)rc eZdZdeffd Zdej deej deejde ej fdZ xZ S)Owlv2ClassPredictionHeadrct||jj}|jj|_t j|j ||_t j|j d|_ t j|j d|_ t j|_ y)Nr) rrrrr query_dimr rr logit_shiftrELUelu)rBrrrs r'rz!Owlv2ClassPredictionHead.__init__s $$00--99ii8 99T^^Q799T^^Q7668r)r6 query_embeds query_maskrc0|j|}|S|j}|jdd\}}tj|||j fj |}||fS|tjj|dddzz }|tjj|dddzz }tjd||}|j|} |j|} |j| dz} || z| z}||jdkDrtj|d }tj|d k(tj |j"j$|}|j tj&}||fS) Nr_rT)rrgư>z...pd,...qd->...pqrrrr)rr!rr$zerosrrrreinsumrrrndimrwherefinforUrcrV) rBr6rrimage_class_embedsr!rr pred_logitsrrs r'rz Owlv2ClassPredictionHead.forwards "[[6  '..F&8&>&>r&B #J ++z;&OPSSTZ[K!34 405<<3D3DEW]_im3D3nqu3uv#u||'8'82W['8'\_c'cd ll#79K\Z &&|4 &&|4 hh{+a/ "[0K?  !""__ZR@ ++jAou{{;CTCT7U7Y7Y[fgK%..7K/00r)) rKrLrMrrr$rOrr rrrrs@r'rrs_ { !1''!1u001!1U\\* !1 u  ! !1r)rceZdZeZdeffd ZedededejfdZ dejdejfdZ e d  d!deded eejdejfd Z d"d ejd ejdedejfdZ d#d ejdeejdeejdeejfdZ d$dejdejdejdeedeededeejfdZ d$dejdeedeededeejf dZ d"dejdejdedejfdZe d%dejdeejdeedeededeedefdZe d%dejdejdeejdeedeededeedefd ZxZS)&Owlv2ForObjectDetectionrct||t||_t ||_t ||_t |d|_tj|jj|jj|_tj|_||_|j"jj$|j"jj&z|_|j"jj$|j"jj&z|_|j-|j(|j*|_y)Nr)rr)rrr-r r class_headrbox_headobjectness_headr rrrr layer_normSigmoidsigmoidrrrnum_patches_heightnum_patches_widthcompute_box_biasbox_biasrs r'rz Owlv2ForObjectDetection.__init__s  ' 26:.v6 5faH,,v';';'G'GVMaMaMpMpqzz|  "&++";";"F"F$++JcJcJnJn"n!%!:!:!E!EIbIbImIm!m--d.E.EtG]G]^ r)rrrcjtjd|dztj}tjd|dztj}tj||d\}}tj||fd}|dxx|zcc<|dxx|zcc<|j dd }|S) Nr)rUxy)indexingrr.r.rr_)r$r%rVmeshgridstackr)rr x_coordinates y_coordinatesxxyybox_coordinatess r'!normalize_grid_corner_coordinatesz9Owlv2ForObjectDetection.normalize_grid_corner_coordinatess Q(9A(=U]]S  Q(:Q(>emmT  }tLB ++r2hB7#44#55*..r15r)rcR|j}|j|}|d}|S)a#Predicts the probability that each image feature token is an object. Args: image_features (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_dim)`)): Features extracted from the image. Returns: Objectness scores. r)detachr)rBrr{s r'objectness_predictorz,Owlv2ForObjectDetection.objectness_predictors4(..0 00@-f5  r)r_)maxsize feature_mapc| td|j||}tj|dd}tj|dztj | dzz }tj |d}|dxx|zcc<|dxx|zcc<tj|dztj | dzz }tj||gd}|S) NzOfeature_map has been deprecated as an input. Please pass in num_patches insteadr"r%g-C6?rrrr)rsrr$cliploglog1p full_liker) rBrrrrbox_coord_biasbox_size box_size_biasrs r'rz(Owlv2ForObjectDetection.compute_box_biass  "no o@@ASUfg**_c3??T#9:U[[/IY\`I`=aa??>37--.. (T/2U[[(TAQ5RR 99nm<"Er) image_featsrc|j|}|r$|j\}}}}|j||}n |j}|j |j }||z }|j |}|S)a Args: image_feats: Features extracted from the image, returned by the `image_text_embedder` method. feature_map: A spatial re-arrangement of image_features, also returned by the `image_text_embedder` method. interpolate_pos_encoding: Whether to interpolate the pre-trained position encodings. Returns: pred_boxes: List of predicted boxes (cxcywh normalized to 0, 1) nested within a dictionary. )rrrrrr!r) rBrrrr|rrrrs r' box_predictorz%Owlv2ForObjectDetection.box_predictor s|&]];/  $:E:K:K 7A!#4a,,-?ARSH}}H;;{112h \\*- r)rrc6|j|||\}}||fS)a8 Args: image_feats: Features extracted from the `image_text_embedder`. query_embeds: Text query embeddings. query_mask: Must be provided with query_embeddings. A mask indicating which query embeddings are valid. )r)rBrrrrrs r'class_predictorz'Owlv2ForObjectDetection.class_predictor+s)-1OOKWa,b)(/00r)rrrrrEc |j||||||d}|rX|j\}}} } | |jjjz} | |jjjz} n|j } |j } |jd} |jjj| }tj|ddddddf|ddddfj}|ddddddf|z}|j|}|jd| | |jdf}|j|}|d}|||fS)NT)rrrrrErrFrrr)r rrrrrrr8ryrsr$ broadcast_torr)rBrrrrrErrrrrrrrKr6class_token_outnew_sizer5s r'image_text_embedderz+Owlv2ForObjectDetection.image_text_embedder?sv**%)/!5%=  $"."4"4 Aq&%!'4;;+D+D+O+O!O  %)B)B)M)M M !%!8!8  $ 6 6 $77:zz..==>OP  ,,\!RaR(-C\RSUXVXUXRXEYE_E_`$Aqr1H-? |4    q !     r "   $++H5 bk \733r)c|jj||d}|rX|j\}}}}||jjj z} ||jjj z} n|j } |j} |d} |jjj| } tj| ddddddf| ddddfj} | ddddddf| z} |j| } | jd| | | jdf}| j|} | |fS)NT)rrrFrrr) r ryrrrrrrrsr$rrr)rBrrrErrrrrrrrKr6rrs r'image_embedderz&Owlv2ForObjectDetection.image_embedderss_00%@Xfj1  $"."4"4 Aq&%!'4;;+D+D+O+O!O  %)B)B)M)M M !%!8!8  $ 6 6 +1-zz..==>OP  ,,\!RaR(-C\RSUXVXUXRXEYE_E_`$Aqr1H-? |4    q !     r "   $++H5 n--r)query_image_featuresquery_feature_mapcj|j|\}}|j|||}t|}g}g} |j} t |j dD]} t jgdg| } || } t| | \}}t j|ddk(r t| | }t j|dz}|d|k\j}|js|| |jd}t j|| d}t j d||}|t j"|}|j%|| || j%|"|r+t j&|}t j&| }nd \}}|||fS) Nr)rrrrr r"g?r)axiszd,id->iNN)rrrr!rBrr$rrprrrwrdnonzeronumelsqueezer#rargminappendr)rBrrrrr}r|pred_boxes_as_cornersbest_class_embedsbest_box_indicespred_boxes_deviceieach_query_boxeach_query_pred_boxesious iou_threshold selected_indsselected_embeddings mean_embedsmean_sim best_box_indr box_indicess r'embed_image_queryz)Owlv2ForObjectDetection.embed_image_querys ../CD<''(<>OQij 8 D188+11!45 6A"\\<.ARSN$9!$< !n.CDGD!yyaC(*>;PQ"IIdOc1M!!W 5>>@M""$&21om6K6KA6N&O##jjaqA  << ;@ST,U\\(-CD !((a)FG '' 5' 6*  ;;'89L++&67K(2 %L+[*44r)query_pixel_valuesrFc ||n|jj}||n|jj}||n|jj}|j ||d}|j ||||\}} |j \} } } } t j|| | | z| f}|j \} } } } t j|| | | z| f}|j|||\}}}|j||\}}|j|||}|s+||||||| jf}td|D}|St||||||d| S)a query_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values of query image(s) to be detected. Pass in one query image per target image. Examples: ```python >>> import requests >>> from PIL import Image >>> import torch >>> from transformers import AutoProcessor, Owlv2ForObjectDetection >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> query_url = "http://images.cocodataset.org/val2017/000000001675.jpg" >>> query_image = Image.open(requests.get(query_url, stream=True).raw) >>> inputs = processor(images=image, query_images=query_image, return_tensors="pt") >>> # forward pass >>> with torch.no_grad(): ... outputs = model.image_guided_detection(**inputs) >>> target_sizes = torch.Tensor([image.size[::-1]]) >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> results = processor.post_process_image_guided_detection( ... outputs=outputs, threshold=0.9, nms_threshold=0.3, target_sizes=target_sizes ... ) >>> i = 0 # Retrieve predictions for the first image >>> boxes, scores = results[i]["boxes"], results[i]["scores"] >>> for box, score in zip(boxes, scores): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected similar object with confidence {round(score.item(), 3)} at location {box}") Detected similar object with confidence 0.938 at location [327.31, 54.94, 547.39, 268.06] Detected similar object with confidence 0.959 at location [5.78, 360.65, 619.12, 366.39] Detected similar object with confidence 0.902 at location [2.85, 360.01, 627.63, 380.8] Detected similar object with confidence 0.985 at location [176.98, -29.45, 672.69, 182.83] Detected similar object with confidence 1.0 at location [6.53, 14.35, 624.87, 470.82] Detected similar object with confidence 0.998 at location [579.98, 29.14, 615.49, 489.05] Detected similar object with confidence 0.985 at location [206.15, 10.53, 247.74, 466.01] Detected similar object with confidence 0.947 at location [18.62, 429.72, 646.5, 457.72] Detected similar object with confidence 0.996 at location [523.88, 20.69, 586.84, 483.18] Detected similar object with confidence 0.998 at location [3.39, 360.59, 617.29, 499.21] Detected similar object with confidence 0.969 at location [4.47, 449.05, 614.5, 474.76] Detected similar object with confidence 0.966 at location [31.44, 463.65, 654.66, 471.07] Detected similar object with confidence 0.924 at location [30.93, 468.07, 635.35, 475.39] ```N)rrr)rrrEr)rrc3&K|] }|| ywrSrQr@xs r'rCzAOwlv2ForObjectDetection.image_guided_detection..2> 1>rJ)r6rrrrr}r7r8)rrrErFrrr$rrrrr>rGr)rBrrrrErrFrrrrrr hidden_dimrquery_image_featsrrrrr}rrs r'image_guided_detectionz.Owlv2ForObjectDetection.image_guided_detectionsv2C1N-TXT_T_TqTq$8$D $++JjJj &1%PSd>dfp1qr HYH_H_E &(9:!MM  ,>AR,RT^_ <@;Q;Q 02J< 8 &(8 '+&:&:{am&:&n#l!..{KIab!! '')F>f>>FM4$0/-%" .  r)c $||n|jj}||n|jj}||n|jj}|j ||||||\}} } | j } | j } | j\} }}}tj| | ||z|f}|jd| z}|j| ||jd}|j| ||jd}|ddkD}|j|||\}}|j|}|j|| |}|s:||||| || j| jf}td|D}|St| |||||| | S)a input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids). output_hidden_states (`bool`, *optional*): Whether or not to return the last hidden state. See `text_model_last_hidden_state` and `vision_model_last_hidden_state` under returned tensors for more detail. Examples: ```python >>> import requests >>> from PIL import Image >>> import torch >>> from transformers import Owlv2Processor, Owlv2ForObjectDetection >>> processor = Owlv2Processor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text_labels = [["a photo of a cat", "a photo of a dog"]] >>> inputs = processor(text=text_labels, images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2] >>> target_sizes = torch.tensor([(image.height, image.width)]) >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> results = processor.post_process_grounded_object_detection( ... outputs=outputs, target_sizes=target_sizes, threshold=0.1, text_labels=text_labels ... ) >>> # Retrieve predictions for the first image for the corresponding text queries >>> result = results[0] >>> boxes, scores, text_labels = result["boxes"], result["scores"], result["text_labels"] >>> for box, score, text_label in zip(boxes, scores, text_labels): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected {text_label} with confidence {round(score.item(), 3)} at location {box}") Detected a photo of a cat with confidence 0.614 at location [341.67, 23.39, 642.32, 371.35] Detected a photo of a cat with confidence 0.665 at location [6.75, 51.96, 326.62, 473.13] ```)rrrrrErrrrc3&K|] }|| ywrSrQr s r'rCz2Owlv2ForObjectDetection.forward..r rJ)r6r5r|rr{r}r7r8)rrrErFrr7r8rr$rrrrr>rGry)rBrrrrrErrFrrrrrrrrr rmax_text_queriesrrr}r{r|rs r'rzOwlv2ForObjectDetection.forward@sh2C1N-TXT_T_TqTq$8$D $++JjJj &1%PSd>dfp1qr %??1-;#++J8H,J\J\]_J`a %%j2BIOOTVDWX v&* '+&:&:; V`&a#l!55kB'' [BZ[ !%%''') F>f>>FM)$$!/%* .  r)rSrrrr})rKrLrMrr:r staticmethodr[r$r rrOrrrrrrrrrrrrrrryrrrs@r'rrsL _{ _cVY^c^j^j !53D3D !IZIZ !qko"%:=LTUZUfUfLg 6*/ &&&&#'     J59-1 1&&1u0011U\\* 1 u  ! 12-1/3).14<<14''14 14 $D> 14 'tn 14#'14 u  !14n-1/3). (.''(.$D>(.'tn (. #' (. u  ! (.^*/ *5#//*5!,,*5#' *5    *5X;?,0/3).&*s ''s %U%6%67s $D> s 'tn s #' s d^s  /s s j 26,0/3).&*r <<r ''r !. r $D> r 'tn r #'r d^r  $r r r)r)r-rrbrwr)BrN dataclassesr functoolsrtypingrrrrr r$torch.utils.checkpointr r activationsr modeling_attn_mask_utilsrrmodeling_outputsrrmodeling_utilsrutilsrrrrrconfiguration_owlv2rrrtransformers.image_transformsr get_loggerrKloggerr(r/r1r\r`rprwryrrnrrrrrrr?rVrbrprwr-rrr__all__rQr)r'r"s9!44 !dK-YYPPF   H %`U\\`ell` -5<<-ELL-  " +"  " LGvG&G EF Ev E" "'0 2 2  2 j - K-  - bJBIIJ\"))@h2RYYh2X ryy  / /d,%?,%,%`V 299V tI 299I Z3 )3 n4 RYY4 p. +. bU %U U rRYY(-1ryy-1`u 2u p rr)