Uh dZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)e%jTe+Z,eGdde#Z-Gdde j\Z/Gdde j\Z0Gdde j\Z1de0iZ2Gdde j\Z3Gdde j\Z4Gdde j\Z5Gd d!e j\Z6Gd"d#e j\Z7e$Gd$d%eZ8Gd&d'e j\Z9Gd(d)e j\Z: dGd*e j\d+e jvd,e jvd-e jvd.ee jvd/eGd4d5e j\Z?Gd6d7e j\Z@Gd8d9e j\ZAe$d:;Gd<d=e8ZBGd>d?e j\ZCe$d@;GdAdBe8ZDe$dC;GdDdEe8eZEgdFZFy)HzPyTorch GIT model.N) dataclass)CallableListOptionalTupleUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging torch_int) GitConfigGitVisionConfigceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y)GitVisionModelOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r"rtorch FloatTensor__annotations__r#r$rr%v/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/git/modeling_git.pyr!r!2sr*15L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r.r!c eZdZdZfdZ d deejdeejdeejde dejf dZ xZ S) GitEmbeddingsz;Construct the embeddings from word and position embeddings.cBt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_t|dd|_|j#dt%j&|jj)ddy) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr6register_bufferr*arangeexpandselfconfig __class__s r/r>zGitEmbeddings.__init__Ss !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej  r. input_idsr8 inputs_embedspast_key_values_lengthreturncJ||j}n|jdd}|d}||jdd|||zf}||j|}n|}|jdk(r|j |}||z }|j |}|j |}|S)Nr:rr7)sizer8rCr6rErFrJ) rPrSr8rTrU input_shape seq_length embeddingsrEs r/forwardzGitEmbeddings.forwardbs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  --i8J&J  ' ': 5"&":":<"H  - -J^^J/ \\*- r.)NNNr) r&r'r(r)r>rr* LongTensorr+intTensorr\ __classcell__rRs@r/r1r1PssE  "153759&' E,,-u//0  1 12  !$   r.r1ceZdZd fd Zdej dej fdZ d dej deejdeejdee d ee d ee de ej fd Z xZ S)GitSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd||_|-tjd|jjd|j|_t|j|jz |_ |j|jz|_ t|jj|jj z dzd z|_|j$|xj"|j$zc_t'j(|j|j|_t'j(|j|j|_t'j(|j|j|_t'j0|j2|_|xs t7|d d |_|j8d k(s|j8d k(rG|j:|_t'j<d|j:zd z |j|_yy)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr6r7 relative_keyrelative_key_query) r=r>rAnum_attention_headshasattr ValueError layer_idxlogger warning_oncerRr&r^attention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingr LinearquerykeyvaluerHattention_probs_dropout_probrJrKr6rDr?distance_embeddingrPrQr6rmrRs r/r>zGitSelfAttention.__init__s(     : : :a ?PVXhHi#F$6$6#78 445Q8 #      !8!8 9:,,  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #>rr.xrVc|jdd|j|jfz}|j|}|j ddddS)Nr:rrgrr )rXrjrpviewpermute)rPr~ new_x_shapes r/transpose_for_scoresz%GitSelfAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FF; yyAq!$$r.r$attention_mask head_maskpast_key_valueoutput_attentionspixel_values_presentc x|j|}|r |jnd}|j|j|} |j|j |} ||j | dddd|dddf| dddd|dddf|j \} } tj| ddddd|ddf| gd} tj| ddddd|ddf| gd} |j|} tj| | jdd}|jdk(s|jdk(r| jd| jd}}|Dtj|dz tj|j j!dd}n@tj"|tj|j j!dd}tj"|tj|j j!dd}||z }|j%||j&zdz }|j)| j* }|jdk(rtj,d | |}||z}nE|jdk(r6tj,d | |}tj,d | |}||z|z}|t/j0|j2z }|||z}t4j6j9|d}|j;|}|||z}tj|| }|j=dddd j?}|jAdd|jBfz}|j!|}|r||fn|f}||fz}|S)Nrrgdimr:rhrirdtypedevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrr )"rxrurryrzupdatermr*catmatmul transposer6shapetensorlongrrrMr|rDtoreinsummathsqrtrpr functionalsoftmaxrJr contiguousrXrq)rPr$rrrrrmixed_query_layercutoff key_layer value_layerkey_layer_pastvalue_layer_past query_layerattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r/r\zGitSelfAttention.forwards!JJ}5,@((a--dhh}.EF // =0IJ  %/=/D/D!Q*+[Avw9I-JDNN0 ,N, 9Q7F7A-=#>"OUVWI))[Aww1A%BDT$U[\]K//0AB !<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L)!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM]^--r.NNNNNFF)r&r'r(r>r*r_rrr+r boolrr\r`ras@r/rcrcs uD%ell%u||%7;15*.,1/4J||J!!2!23JE--. J ! J $D> J'tnJ u|| Jr.rccneZdZfdZdej dej dej fdZxZS) GitSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr4) r=r>r rwrAdenserFrGrHrIrJrOs r/r>zGitSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r.r$ input_tensorrVcr|j|}|j|}|j||z}|SNrrJrFrPr$rs r/r\zGitSelfOutput.forward7 =1  ]3 }|'CD r.r&r'r(r>r*r_r\r`ras@r/rr1> U\\RWR^R^r.reagerceZdZd fd ZdZ d dej deejdeejdee dee dee d e ej fd Z xZ S) GitAttentionct|t|j||||_t ||_t|_y)N)r6rm) r=r>GIT_SELF_ATTENTION_CLASSES_attn_implementationrProutputset pruned_headsr}s r/r>zGitAttention.__init__ sE .v/J/JK ,Cy  $F+ Er.c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrrPrjrprrrxryrzrrrqunion)rPheadsindexs r/ prune_headszGitAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r.r$rrrrrrVcn|j||||||}|j|d|}|f|ddz} | S)Nrr)rPr) rPr$rrrrr self_outputsattention_outputrs r/r\zGitAttention.forward%sTyy        ;;|A F#% QR(88r.rr)r&r'r(r>rr*r_rr+r rrr\r`ras@r/rr s";*7;15*.,1/4||!!2!23E--.  !  $D> 'tn u|| r.rcVeZdZfdZdej dej fdZxZS)GitIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r=r>r rwrAintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrOs r/r>zGitIntermediate.__init__=s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r.r$rVcJ|j|}|j|}|Sr)rrrPr$s r/r\zGitIntermediate.forwardEs& =1 00? r.rras@r/rr<s#9U\\ellr.rcneZdZfdZdej dej dej fdZxZS) GitOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r=r>r rwrrArrFrGrHrIrJrOs r/r>zGitOutput.__init__Ms` YYv779K9KL f&8&8f>S>STzz&"<"<= r.r$rrVcr|j|}|j|}|j||z}|Srrrs r/r\zGitOutput.forwardSrr.rras@r/rrLrr.rceZdZd fd Z d dej deejdeejdeedee dee de ej fd Z d Z xZ S) GitLayerct||j|_d|_t |||_t ||_t||_ y)Nr)rm) r=r>chunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rPrQrmrRs r/r>zGitLayer.__init__[sK '-'E'E$%f B+F3' r.r$rrrrrrVc|j||||||}|d}|dd} |d} t|j|j|j|} | f| z} | | fz} | S)N)rrrrrr:)rrfeed_forward_chunkrr) rPr$rrrrrself_attention_outputsrrpresent_key_value layer_outputs r/r\zGitLayer.forwardcs"&   /)!5 "0" 2!4)2.2260  # #T%A%A4CSCSUe  /G+.00r.cL|j|}|j||}|Sr)rr)rPrintermediate_outputrs r/rzGitLayer.feed_forward_chunks,"//0@A{{#68HI r.rr)r&r'r(r>r*r_rr+r rrr\rr`ras@r/rrZs(7;15*.,1/4 || !!2!23 E--.  !  $D> 'tn  u||  Dr.rceZdZfdZ d dej deejdeejdeee e e ejfdee dee dee d ee d ee d ee ej e ffd Z xZS) GitEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}wNF) r=r>rQr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rPrQirRs r/r>zGitEncoder.__init__sP  ]]vG_G_A`#aAHVQ$7#ab &+#$bsA$r$rrpast_key_values use_cacheroutput_hidden_statesr return_dictrVc |jr%|jr|rtjdd}d} |rIt |t s9d} | t }n*t j|}tjd|rdnd} |rdnd} d} t|jD]t\}}|r| |fz} |||nd}|jr-|jr!|j|j|||||}n |||||||}|d}|r|d} |sl| |dfz} v|r| |fz} |r| nd}| r|j}| std ||| | fDSt||| | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)r-rr:rc3$K|]}|| ywrr-.0vs r/ z%GitEncoder.forward..s  = sr#rr$r%)rtrainingrnrorr r from_legacy_cache enumerater_gradient_checkpointing_func__call__to_legacy_cachetupler)rPr$rrrrrrrrreturn_legacy_cacheall_hidden_statesall_self_attentionsnext_decoder_cacher layer_modulelayer_head_mask layer_outputs next_caches r/r\zGitEncoder.forwards  & &4==##p" $ Z?"& &".."."@"@"Q##^ #7BD$5b4!(4 POA|#$58H$H!.7.CilO**t}} $ A A ))!"##% ! !-!"##%( ! *!,M%22%6" &9]1=M 1]4D D +4'$ #335J "%'    '+&+*   r.)NNNNFFFT)r&r'r(r>r*r_rr+rr rrrr\r`ras@r/rrs,7;15SW$(,1/4/4&*Z ||Z !!2!23Z E--. Z "%uU5;L;L5M/N(N"OP Z D> Z $D>Z 'tnZ 'tnZ d^Z  uU\\"$;; <Z r.rc&eZdZeZdZdZdZdZdZ y)GitPreTrainedModelgitTct|trtjj |j d|j jtjj |jj|j jtjj |jj|j jt|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j 2|jj|j jyyt|tj"rJ|jjj|jjj%dyy)zInitialize the weights)meanstd)rNg?)rGitVisionEmbeddingsr initnormal_class_embeddingrQinitializer_rangepatch_embeddingweightposition_embeddingrwdatabiaszero_r?r3rFfill_)rPmodules r/ _init_weightsz GitPreTrainedModel._init_weightss f1 2 GGOOF22$++B_B_O ` GGOOF2299t{{?\?\O ] GGOOF55<<$++B_B_O ` fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r.N) r&r'r(r config_classbase_model_prefixsupports_gradient_checkpointing_supports_cache_class_supports_quantized_cacher*r-r.r/rrs$L&*#  $*r.rceZdZdeffd Zdej dededej fdZd dejdej fd Z xZ S) rrQct|||_|j|_|j |_|j |_tjtj|j|_ tj|j|j|j |j d|_|j |j zdz|_|jdz|_tj"|j |j|_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestrider&rgrr8r9r;)r=r>rQrA embed_dimrsrtr Parameterr*randnr Conv2d num_channelsr" num_patches num_positionsr?r$rLrMrNrOs r/r>zGitVisionEmbeddings.__init__s   ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr.r[heightwidthrVc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr:g?r rgbicubicF)rXmode align_cornersr)rr$r# unsqueezer*jit is_tracingr8rtrreshaperr r interpolaterr) rPr[r=r>r;r$r<class_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r/interpolate_pos_encodingz,GitVisionEmbeddings.interpolate_pos_encoding$sv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr. pixel_valuesc `|j\}}}}|sJ||jk7s||jk7r,td|d|d|jd|jd |jjj }|j|j |}|jdjdd}|jj|dd} tj| |gd } |r| |j| ||z} | S| |j|jz} | S) NzInput image size (*z) doesn't match model ().rrgrr:r)rrsrlr"r#rrflattenrr rNr*rrMr$r8) rPrNrM batch_size_r=r> target_dtype patch_embeds class_embedsr[s r/r\zGitVisionEmbeddings.forwardMs6'3'9'9$ Avu'Vt-F%SWSbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr.F) r&r'r(rr>r*r_r^rMr+r\r`ras@r/rr sdqq,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr.rcVeZdZfdZdej dej fdZxZS) GitVisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r=r>rQr r activation_fnr rwrArfc1fc2rOs r/r>zGitVisionMLP.__init__asd  #F$5$5699V//1I1IJ99V55v7I7IJr.r$rVcl|j|}|j|}|j|}|Sr)r]r\r^rs r/r\zGitVisionMLP.forwardhs4/ **=9 / r.rras@r/rZrZ`s$KU\\ellr.rZr)rxryrzrscalingrJc tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)Nr:r)rr)prrrg) r*rrr rrfloat32rrrJrr) r)rxryrzrr`rJkwargs attn_weights attn_outputs r/eager_attention_forwardrgps<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r.ceZdZdZfdZ d dej deej deej deede ej eej ff dZ xZ S) GitVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rQgF)r=r>rQrAr6rj num_headshead_dimrlscaleattention_dropoutrJ is_causalr rwk_projv_projq_projout_projrOs r/r>zGitVisionAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r.r$rcausal_attention_maskrrVc |j\}}}|j|}|j|} |j|} |j |||j |j jdd}| j |||j |j jdd} | j |||j |j jdd} |jjdk7r||||z}n| |}n |du|_ t} |jjdk7rN|jjdk(r|rtjdnt|jj} | ||| | ||j|j|j sdn |j" \} } | j%|||j'} |j)| } |sd} | | fS) z#Input shape: Batch x Time x Channelrrgflash_attention_2Nrsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)ror`rJ)rrrrprqrrkrlrrQrrorgrnrorrmrrJrFrrs)rPr$rrtrrSrZr6querieskeysvaluesattention_interfacerfres r/r\zGitVisionAttention.forwards-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc ;; + +/B B).C.O!/2G!G&2!62$>DN(? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0  LL((r.)NNF) r&r'r(r)r>r*r_rrrr\r`ras@r/riris}GB.268<,1 5)||5)!.5) ( 5 5) $D> 5) u||Xell33 4 5)r.ric eZdZdeffd Z d dej dej dej deede ejf dZ xZ S) GitVisionEncoderLayerrQcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yr) r=r>rAr6ri self_attnr rFrG layer_norm1rZmlp layer_norm2rOs r/r>zGitVisionEncoderLayer.__init__sm +++F3<<Fr*r_rrrr+r\r`ras@r/r}r}sfSS-2 &||& & %|| & $D> & u  ! &r.r}ceZdZdZdeffd Z d deejdeejdee dee dee d e e e ff d Z xZS) GitVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rQct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) r=r>rQr rrrr}layersr)rPrQrTrRs r/r>zGitVisionEncoder.__init__sP  mmERXRjRjLk$lq%:6%B$lm &+#%msA#rrtrrrrVc||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|} t |j D]b\} } |r|| fz}|j r,|jr |j| j| |||} n | | |||} | d} |sZ|| dfz}d|r|| fz}|std| ||fDSt| ||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr-)rrrc3&K|] }|| ywrr-rs r/rz+GitVisionEncoder.forward..dseqWXWdesr#r$r%) rQrruse_return_dictr rrrr r r r) rPrTrrtrrrencoder_statesall_attentionsr$idx encoder_layerrs r/r\zGitVisionEncoder.forwardsHL2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r.)NNNNN)r&r'r(r)rr>rr*r_rrrrr\r`ras@r/rr s,,268<,0/3&*O !.O  ( 5 O $D> O 'tn O d^O  uo% &O r.rceZdZdeffd Ze d deejdee dee dee dee de e e ff d Z xZS) GitVisionTransformerrQc t|||_|j}t ||_t j||j|_ t||_ t j||j|_ yr) r=r>rQrArr[r rFrG pre_layrnormrencoderpost_layernorm)rPrQr6rRs r/r>zGitVisionTransformer.__init__lsj  && -f5LL8M8MN'/  ll9&:O:OPr.rNrrrMrrVc||n|jj}||n|jj}||n|jj}| t d|j ||}|j |}|j||||}|d}|j|}|s |f|ddzSt||j|jS)Nz You have to specify pixel_valuesrM)rTrrrrrr) rQrrrrlr[rrrrr$r%) rPrNrrrMrr$encoder_outputsr#s r/r\zGitVisionTransformer.forwardvs2C1N-TXT_T_TqTq$8$D $++JjJj &1%rrr*r+rrrrr\r`ras@r/rrjsQQ59,0/338&* & u001& $D>& 'tn & #+4. & d^ &  uo% && & r.rzY The vision model from CLIP, used in GIT, without any head or projection on top. ) custom_introceZdZeZdZdeffd ZdejfdZ e d de e jde ede eded e edeeeff d ZxZS) GitVisionModelrNrQcdt||t||_|j yr)r=r>r vision_model post_initrOs r/r>zGitVisionModel.__init__s'  08 r.rVcB|jjjSr)rr[r"rPs r/get_input_embeddingsz#GitVisionModel.get_input_embeddingss  ++;;;r.rrrMrcb||n|jj}|j|||||S)a{ Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```)rNrrrMr)rQrr)rPrNrrrMrs r/r\zGitVisionModel.forwardsA8&1%r Modulerrrr*r+rrrrr\r`ras@r/rrs #L$O # 'tn # #' # d^ #  uo% &# # r.rc\eZdZdeffd Zdej dej fdZxZS) GitProjectionrQc0t|||_tjtj |j j|jtj|j|j j|_ yr) r=r>rQr SequentialrwrrrArFrGvisual_projectionrOs r/r>zGitProjection.__init__sf  !# IIf**668J8J K LL++1E1E1T1T U" r.r[rVc$|j|Sr)r)rPr[s r/r\zGitProjection.forwards%%j11r.) r&r'r(rr>r*r_r\r`ras@r/rrs* y 2%,,25<<2r.rzy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states ceZdZfdZdZdZdZdedejdejdejfd Z dd Z e dd eejd eejd eejdeejdeejdeejdeeeeej&fdeedeedeededeedeeejeffdZxZS)GitModelclt||_t|_t j |_t|_ t|_ j6tjfdtjD|_|j#y)Nc3K|]B}tjtjddjj Dyw)rN)r r7r*zerosrrrA)rrTrQs r/rz$GitModel.__init__..s;; U[[Av/C/C/O/OPQ;sAA )r=r>rQr1r[rrr image_encoderrrrrrvr ParameterListrimg_temperal_embeddingrrOs `r/r>zGitModel.__init__s   '/+F,@,@A!&) !.v!6  * * 6*,*:*:;v>>?;+D ' r.c.|jjSrr[rCrs r/rzGitModel.get_input_embeddingss...r.c&||j_yrr)rPrzs r/set_input_embeddingszGitModel.set_input_embeddingss*/'r.c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rPheads_to_prunerrs r/ _prune_headszGitModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr.rXrrrVctjtj||||d}|j|dk(t d}|S)Nrrr)diagonal-inf)r*triuones masked_fillfloat)rPrXrrmasks r/_generate_future_maskzGitModel._generate_future_masksAzz%**T4eLWXY 5=9 r.c|jd}|jd}|j}|j} tj||f|| } tj |||zft d|j| } tj||f| |j} |dkDrAtj|jd|jd|zf| |j}tj| | fd} tj| |j| fd}tj| |fddddf}|4tj |jd|jdfd|}|jtjk7r td tj||j }t d||<|j|jd||z||z|zf}|j}|ddddd|f}|dddddf}||z|ddddd|f<|dddddddf}|S) NrrrrrrF) fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrrr*rfullrrrrrl zeros_likerNclone)rPtgtmemorytgt_maskrUmemory_key_padding_masknum_tgt num_memoryrrtop_left top_right bottom_leftleftrightfull_attention_maskzero_negative_infinity origin_leftrs r/create_attention_maskzGitModel.create_attention_masksF))A,\\!_  ;; J7eTJJ #99 : &M::  kk j !?? "A %{{"HNN1$58N$NOH yy(K0a8 9hkk%&89qA#iiu 1=dAgF " *&+jj&,,q/6<>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer:z5You have to specify either input_ids or inputs_embedsrrrgrrz#pixel_values must be of rank 4 or 5)rSr8rTrUr)rrrrU)tgt_len)rrrrrrrrr)%rQrrrrrl%warn_if_padding_and_no_attention_maskrXrr rget_seq_length get_head_maskrndimrr#rrappendr*rrr[rrrrepeatrrrrrrrr$r%)rPrSrr8rNrrTrrrrrMrrYrZrUprojected_visual_featuresvisual_features frame_idxvisual_features_frameembedding_outputr$rcombined_attention_maskexpanded_attn_maskrsequence_outputs r/r\zGitModel.forwardFsCJ2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y. Q#..*K  &',,.s3KTU U ^ "#  &"/59 "1%++A.$335 #&&y$++2O2OP $(!  #  A%"&"4"4 ;S#5### ""a'"$!&|'9'9!',>$Q 1a%78Sk-?-''**T-H-H-SS)#**+@A B#())O"C!!FGG(,(>(>(O %??%'#9 +  % ,(- !''*A/?/E/Ea/HI&,,'..) %%>$D$D  ! !! $(A(F(Fq(I I1a% !  #<>N"OUVW --j:J:P:PRbRiRij#'"<"< ,#9 #=#   %"< 0 6 6 B"b!(() &)%71?U>U>VXY8Y%Z"'1{1~o.?+a.AR(RSWiiS,, 2+/!5#!-T!9'  *!,#%(;; ;&-+;;)77&11   r.r) NNNNNNNNNNFN)r&r'r(r>rrrr^r*rrr_rrrrrr rr+rrrr\r`ras@r/rrs &/0C#ekk5<<\a\h\h 0#d-115/3/3,004KO$(,0/3).&*c ELL)c !.c u||, c u||, c ELL) c   -c "%tE4E4E/F(F"GHc D>c $D>c 'tnc #'c d^c  uU\\"$>> ?c c r.rz` GIT Model with a `language modeling` head on top for autoregressive language modeling. c eZdZdgZfdZdZdZe ddee jdee jdee jdee jd ee jd ee jd ee jd ee e e e jfd eedeedeededeede ee jeffdZ ddZdZxZS)GitForCausalLMz output.weightct||t||_t j |j |j|_|jyr) r=r>rrr rwrAr@rrrOs r/r>zGitForCausalLM.__init__sF  F#ii 2 2F4E4EF  r.c|jSrrrs r/get_output_embeddingsz$GitForCausalLM.get_output_embeddingss {{r.c||_yrr)rPnew_embeddingss r/set_output_embeddingsz$GitForCausalLM.set_output_embeddingss $ r.rSrr8rNrrTlabelsrrrrrMrrVc | | n|jj} |d} |j|||||||| | | | |  }|d}|j|}d}||jjj dj jj}|dd|dddfj}|ddddfj}|j|jd|jj|jdfd|jji|}| s|f|ddz}||f|zS|St|||j|j|j S) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) rr8rNrrTrrrrrMrrr:rr@)losslogitsrr$r%)rQrrrrrrrPrur loss_functionrr@rrr$r%)rPrSrr8rNrrTrrrrrrMrrdrrrrnum_image_tokensshifted_logitsrs r/r\zGitForCausalLM.forwardsj&1%IIKNAqrE]--/F%4%%##B (>(>? B ;;11 DY,F)-)9TGf$ Ev E%#33!//))   r.c |B|j}|jd|kDr|}n|jddz }|dd|df}|j}||j|}|||jdd||dS)NrrN)rSrrNrr)rrnew_onesget) rPrSrrrrd past_lengthremove_prefix_lengthrYs r/prepare_inputs_for_generationz,GitForCausalLM.prepare_inputs_for_generations  &)88:Kq!K/'2$(1q'9A'=$!!%9%:":;I oo  !&// .s.nU_j--aZ=N=N1OPns58)r )rPrrreordered_past layer_pasts ` r/_reorder_cachezGitForCausalLM._reorder_caches=) J ncmnn N r.) NNNNNNNNNNNFN)NNN)r&r'r(_tied_weights_keysr>rrrrr*r_rr rrrrr\rr r`ras@r/rrs **%-115/3/3,004)-FJ$(,0/3).&*A ELL)A !.A u||, A u||, A ELL) A   -A &A "%tELL/A(A"BCA D>A $D>A 'tnA #'A d^A uU\\"$:: ;!A A HOS >r.r)rrrr)r)Gr)r dataclassesrtypingrrrrrr*torch.utils.checkpointr activationsr cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_outputsrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrconfiguration_gitrr get_loggerr&rnr!rr1rcrrrrrrrrrrZr_rrgrir}rrrrrr__all__r-r.r/rs  !99 !.)B Gll :   H % ?;? ?8-BII-`rryyrl BII   /299/f bii   .ryy.ba a H***:P"))Pf 299 .% II% <<% % << % U\\* %  %%,L)L)`/BII/f^ ryy^ B3 2993 l 2 '2  2 j 2BII 2 ~ !~  ~ B {'{  {| Qr.