Uh`ddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZmZmZdd lmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'ddl(m)Z)m*Z*e$jVe,Z-edGdde j\Z/ dAde j\de j`de j`de j`dee j`de1de1fdZ2Gdde j\Z3e!GddeZ4eGd d!eZ5Gd"d#e j\Z6Gd$d%e j\Z7Gd&d'e j\Z8e jre/d(Z:Gd)d*e j\Z;Gd+d,e j\ZGd1d2e j\Z?eGd3d4eZ@e!d56Gd7d8e>ZAeGd9d:e ZBGd;dd?e>eZDgd@ZEy)BN) dataclass)CallableListOptionalTupleUnion)ACT2FN)GenerationMixin)use_kernel_forward_from_hub)FlashAttentionKwargs)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargs ModelOutputauto_docstringcan_return_tupleis_torchdynamo_compilinglogging torch_int) AutoModel)InternVLConfigInternVLVisionConfigRMSNormc,eZdZdfd ZdZdZxZS)InternVLVisionRMSNormct|tjt j ||_||_y)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s /var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/internvl/modeling_internvl.pyr%zInternVLVisionRMSNorm.__init__7s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetor(float32powmeanrsqrtr+r*)r, hidden_states input_dtypevariances r0forwardzInternVLVisionRMSNorm.forward?sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r1c^t|jjd|jS)Nz, eps=)tupler*shaper+r,s r0 extra_reprz InternVLVisionRMSNorm.extra_reprFs* ))*+6$2G2G1HIIr1)gư>)__name__ __module__ __qualname__r%r>rC __classcell__r/s@r0r"r"5s$;Jr1r"modulequerykeyvalueattention_maskscalingdropoutc |}|} tj||jdd|z} |#|ddddddd|jdf} | | z} tj j | d} tj j| ||j} tj| | } | jddj} | | fS)Nrr r3dim)ptrainingr) r(matmul transposerAr& functionalsoftmaxrOrU contiguous) rIrJrKrLrMrNrOkwargs key_states value_states attn_weights causal_mask attn_outputs r0eager_attention_forwardraJsJL<<z';';Aq'ABWLL!$Q1.D 0@0@0D.D%DE #k1 ==((2(>L==((6??([L,,|\:K''1-88:K  $$r1c eZdZdZdeffd Z d dejdeejdeejde e fdZ xZ S) InternVLVisionAttentionz+Attention Class for InternVL Vision Encoderconfigct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ |j}|j}d|_ tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j|_|dkDrtj*|ntj,|_ |rt/|jntj,|_|rt/|j|_ytj,|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r$r%rdr- embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr&Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityr"q_normk_norm)r,rd proj_dropoutqk_normr/s r0r%z InternVLVisionAttention.__init__hs  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!900 $$ii0NU[UjUjk ii0NU[UjUjk ii0NU[UjUjk " $..$.. I>JQ>N"**\":TVT_T_Ta?F+DNN;BKKM ?F+DNN; BKKM r1r;rMoutput_attentionsr[c $|j\}}}|j|}|j|} |j|} |j |}|j | } |j |||j|jjdd}| j |||j|jjdd} | j|||j|jjdd} t} |jjdk7r^|jjdk(r(|jddrtj!dnt"|jj} | ||| | |f|j$sdn |j&|j(dd |\} } | j |||j*} |j-| }|j/|}|r|| f}|S|df}|S) Nrreagersdpar~Fz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rOrNrq)sizertrurvrzr{reshaperjrkrWviewrard_attn_implementationgetlogger warning_oncerrUrnrmrhrwro)r,r;rMr~r[ batch_sizeseq_len_ query_statesr\r]attention_interfacer`r^outputoutputss r0r>zInternVLVisionAttention.forwards"/!3!3!5 GQ{{=1 [[/ {{=1 {{<0 [[, #++JQUQ^Q^_iijkmno '' GT^^T]][eefgijk #((Wdnndmm\ffghjkl (? ;; + +w 6{{//69fjjI\^c>d##L '>dkk>^>^&_#$7      % $}}C$2H2HJJ %  % ! \"))*gt~~N &&{3((0,=6<(EKD>r1)NN) rDrErF__doc__rr%r(Tensorrrr r>rGrHs@r0rcrcesd5Z3Z>2648 /||/!./$ELL1 / -. /r1rcc0eZdZeZdZdZdZdgZdZ dZ dZ y)InternVLVisionPreTrainedModelinternvl_vision pixel_valuesTInternVLVisionLayercrt|tjtjtjfrm|j j jd|jj|j%|jj jyyt|tjrz|j j jd|jj|j2|j j |jjyyt|tjrJ|jj j|j j jdyt|t r|j"j j|j$$|j$j j|j&%|j&j jyyt|t(rs|j*j j|jj,|j.j j|jj,yy)zInitialize the weightsrr9stdN?) isinstancer&rrConv2dConvTranspose2dr*datanormal_rdinitializer_rangergzero_ Embedding padding_idx LayerNormfill_InternVLVisionEmbeddings cls_token mask_tokenposition_embeddingsrlambda_1layer_scale_init_valuelambda_2)r,rIs r0 _init_weightsz+InternVLVisionPreTrainedModel._init_weightss fryy"))R5G5GH I MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S )  8 9    ! ! ' ' )  ,!!&&,,.))5**//5576  3 4 OO & &t{{'I'I J OO & &t{{'I'I J5r1N) rDrErFr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_2rr1r0rrs2'L)$O&*#./N!Kr1rceZdZdZy)$InternVLVisionModelOutputWithPoolinga Class for outputs of [`InternVLVisionModel`]. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N)rDrErFrrr1r0rrsr1rcZeZdZdZfdZdej dej fdZxZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c^t||j|j}}|j|j }}|d|dz|d|dzz}|d|dz|d|dzf}||_||_||_||_||_tj|||||_ y)Nrr) kernel_sizestride) r$r% image_size patch_size num_channelsr- num_patches patch_shaper&r projection) r,rdrrrr-rrr/s r0r%z&InternVLVisionPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk !!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&))L+:^hir1rreturnc|j\}}}}||jk7r td|j|}|jd|jd}}|j dj dd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rr r)rArrlrflattenrW) r,rrrheightwidth embeddings patch_height patch_widths r0r>z%InternVLVisionPatchEmbeddings.forward s2>2D2D/ L&% 4,, ,w __\2 $.$4$4Q$79I9I!9Lk ''*44Q: L+666r1) rDrErFrr%r(rr>rGrHs@r0rrs) j 7ELL 7U\\ 7r1rceZdZdZdeddffd Zdejdededejfd Z d d ejd e ejdejfd Z xZ S)rzc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rdrNc2t|tjt j dd|j |_|jr:tjt j dd|j |_ nd|_ t||_ |j|_ t|jtj j"r |jn|j|jf|_|jj$}|j&r=tjt j d|dz|j |_nd|_tj*|j,|_y)Nr)r$r%r&r'r(zerosr-ruse_mask_tokenrrpatch_embeddingsrrr collectionsabcIterabler use_absolute_position_embeddingsrrxhidden_dropout_probrO)r,rdrr/s r0r%z!InternVLVisionEmbeddings.__init__ s$ ekk!Q8J8J&KL   ll5;;q!V=O=O+PQDO"DO =f E ++&++[__-E-EF   ##V%6%67  ++77  2 2')||EKK;QR?TZTfTf4g'hD $'+D $zz&"<"<= r1rrrc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j dz} ||j dz} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr3r?r rbicubicF)rmode align_cornersrR)rArr(jit is_tracingrrrpermuter&rX interpolatercat) r,rrrr num_positionsclass_pos_embedpatch_pos_embedrS new_height new_widthsqrt_num_positionss r0interpolate_pos_encodingz1InternVLVisionEmbeddings.interpolate_pos_encoding6sj!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"tq11 T__Q// &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr1rbool_masked_posc|j\}}}}|j|\}\}}|j\} } }|K|jj | | d} |j dj | } |d| z z| | zz}|jj | dd} tj| |fd}|j||j|||z}|j|}|||ffS)Nr3rrR) rArrrexpand unsqueezetype_asrr(rrrrO)r,rrrrrrrrrr mask_tokensw cls_tokenss r0r>z InternVLVisionEmbeddings.forward^s +001fe262G2G 2U/ /\;!+!2 GQ  &//00WbIK))"-55kBA#q1u- a?J^^**:r2> YY J7Q?  # # /#d&C&CJPVX]&^^J\\*- L+666r1N)rDrErFrrr%r(rintrr BoolTensorr>rGrHs@r0rrs >3>>,&D5<<&D&DUX&D]b]i]i&DV7;7ll7"%"2"237  7r1rcVeZdZfdZdej dej fdZxZS)InternVLVisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r$r%rdr hidden_act activation_fnr&rrr-intermediate_sizefc1fc2r,rdr/s r0r%zInternVLVisionMLP.__init__ysd  #F$5$5699V//1I1IJ99V55v7I7IJr1r;rcl|j|}|j|}|j|}|Sr)rrr)r,r;s r0r>zInternVLVisionMLP.forwards4/ **=9 / r1)rDrErFr%r(rr>rGrHs@r0rrxs$KU\\ellr1r) layer_normrms_normc eZdZdZdeddffd Z d dejdede e eje ejejfffdZ xZ S) rz?This corresponds to the Block class in the timm implementation.rdrNct||j|_d|_t ||_t ||_t|j|j|j|_ t|j|j|j|_ |j}tj |t#j$|jzd|_tj |t#j$|jzd|_tj*|j,|_y)Nrr.T) requires_grad)r$r%chunk_size_feed_forward seq_len_dimrc attentionrmlpNORM2FN norm_typer-layer_norm_epslayernorm_beforelayernorm_afterrr&r'r(r)rrrxrrO)r,rd init_valuesr/s r0r%zInternVLVisionLayer.__init__s '-'E'E$08$V, '(8(8 9&:L:LRXRgRg h&v'7'789K9KQWQfQfg33  [5::v?Q?Q3S%Scgh  [5::v?Q?Q3S%Scgh zz&"<"<= r1r;r~c"|j|j||\}}|j|z}||z}|j|}|j |}|j |}|j |j |z}||z}||fS)N)r~)rr rr rrOr)r,r;r~attention_outputattention_weights layer_outputs r0r>zInternVLVisionLayer.forwards /3nn  ! !- 0//=/ ++  ==+;;)=8 ++M: xx - ||L1 == $==<7L$m3 ...r1)F) rDrErFrrr%r(rboolrrr>rGrHs@r0rrsiI >3 > >$#(/||/ / uU\\"E%,, *D$EE F /r1rc neZdZdeddffd Ze d dejdedede e e ffdZ xZ S) InternVLVisionEncoderrdrNct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r$r%rdr& ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r,rdir/s r0r%zInternVLVisionEncoder.__init__sO  ]]vOgOgIh#iA$7$?#ij &+#$jsA#r;r~output_hidden_statesc0|rdnd}|rdnd}t|jD]]\}}|r||fz}|jr*|jr|j |j ||}n |||}|d}|sU||dfz}_|r||fz}t |||S)Nrrrlast_hidden_stater; attentions) enumeraterrrU_gradient_checkpointing_func__call__r) r,r;r~rall_hidden_statesall_self_attentionsr layer_module layer_outputss r0r>zInternVLVisionEncoder.forwards#7BD$5b4(4 POA|#$58H$H!**t}} $ A A ))=:K! !-]rGrHs@r0rrsg,3,, #(%*  ||     #  uo% &    r1rceZdZdeddffd ZdZee d dejde ejde e d e e de eeff d ZxZS) InternVLVisionModelrdrNc2t||||_t||_t ||_|jrtjn*tj|j|j|_ |jy)Nr)r$r%rdrrrencoderuse_mean_poolingr&ryrr-r layernorm post_initrs r0r%zInternVLVisionModel.__init__so   26:,V4 $44BKKM",,vGYGY_e_t_t:u  r1c.|jjSr)rrrBs r0get_input_embeddingsz(InternVLVisionModel.get_input_embeddingss///r1rrr~rc.||n|jj}||n|jj}|j||\}}|j |||}|d}|j |}t ||j|jS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )r)r~rrr) rdr~rrr)r+rr;r) r,rrr~rembedding_outputrencoder_outputssequence_outputs r0r>zInternVLVisionModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj #oolOo\!,, /!5'  *!,..93-)77&11  r1)NNN)rDrErFrr%r.rrr(rrrrrr@rr>rGrHs@r0r'r's 3  07;,0/3  ll "%"2"23 $D>  'tn  u:: ;   r1r'c:eZdZeZdZdZdZdZdZ dZ dZ dZ dZ dZy)InternVLPreTrainedModelTpast_key_valuesc$t|jd|jjj}t |t j rY|jjjd||j%|jjjyyt |t jrJ|jjj|jjjdyy)Nrrrr)getattrrdget_text_configrrr&rrr*rrrgrrr)r,rIrs r0rz%InternVLPreTrainedModel._init_weights(sdkk#6 8S8S8U8g8gh fbii ( MM   & &CS & 9{{&   &&('  - KK   " " $ MM   $ $S ).r1N)rDrErFrrrr_skip_keys_device_placement_supports_cache_classrr_supports_quantized_cache_supports_static_cache_supports_attention_backendrrr1r0r4r4sA!L&*#"3 !N $!"& *r1r4c*eZdZdeffd ZdZxZS)InternVLMultiModalProjectorrdc*t|tj|jj t d|jz dzz|_tj|jj t d|jz dzz|jj |_ t|j|_tj|jj |jj |_y)Nrr)r$r%r&r vision_configr-rdownsample_ratiorrr text_configlinear_1r projector_hidden_actactlinear_2rs r0r%z$InternVLMultiModalProjector.__init__5s ,,v';';'G'G#aRXRiRiNiJjnoJo'op  , ,s1v7N7N3N/OST/T TV\VhVhVtVt  &556 &"4"4"@"@&BTBTB`B`a r1c|j|}|j|}|j|}|j|}|Sr)rrErGrH)r,image_featuresr;s r0r>z#InternVLMultiModalProjector.forward>s@7  m4 /  m4 r1)rDrErFrr%r>rGrHs@r0r@r@4sb~br1r@c:eZdZUdZdZeejed<y)InternVLModelOutputWithPasta Base class for InternVL outputs, with hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) rDrErFrrMrr( FloatTensor__annotations__rr1r0rLrLFs88<%"3"34;r1rLzx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. ) custom_introc%eZdZddiZdeffd ZdZdZdejde e e e fd e fd Zee dd ej"dejd eej&d eej"dee ejdeejdee e e e fd ee deedeedeedeedeej"dej&deede eeff dZddej&defdZxZS) InternVLModelzlanguage_model.modellanguage_modelrdct||tj|j|_t ||_tj|j|_ |jyr) r$r%r from_configrB vision_towerr@multi_modal_projectorrDrSr,rs r0r%zInternVLModel.__init__osY  %11&2F2FG%@%H"'33F4F4FG r1c6|jjSr)rSr.rBs r0r.z"InternVLModel.get_input_embeddingsws""7799r1c:|jj|yr)rSset_input_embeddingsr,rLs r0rZz"InternVLModel.set_input_embeddingszs 007r1rvision_feature_layervision_feature_select_strategyc |jj}|dk(r|j|j}n|j |j |}|dk(r|ddddddf}|j d}t|dz}|j d} |j| ||d}|j||}|j| d|j d}|j|}|S) a% Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `List[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. r3)rdefaultNrrr) scale_factor) rdrCrVr vision_modelr;rArr pixel_shufflerW) r,rr\r]r[rCvision_featureschannels feature_sizers r0get_image_featuresz InternVLModel.get_image_features}s $ ;;77 2 %"//\/J\\O"//\/JXXYmnO )Y 6-aQh7O#((+8S=) $**1- *11*lLZ\],,_K[,\*11*b/BWBWXZB[\44_Er1 input_idsrM position_idsr6 inputs_embeds use_cacher~r return_dictcache_position image_sizesr[rc | | n|jj} | | n|jj} | | n|jj} ||n|jj}||n|jj }|du|duz r t d||j|}||j||||}|||jtj|jjtj|jk(}|jdjdd}ny||jjk(jd}|j!|j#|j}||jjk(j}t%s{||j'|j'k7rW||jjk(j}|j(d|j(dz}t d|d ||j#|j|j*}|j-||}|j.d ||||| | | d | d |}t1|j2|j4|j6|j8| Sd S)Nz:You must specify exactly one of input_ids or inputs_embeds)rr\r]rm)r5devicerrRrr3z6Image features and image tokens do not match: tokens: z , features T) rMrhr6rirjr~rrkrl)rr6r;rrMr)rdr~ruse_return_dictr\r]rlr.rfr(tensorimage_token_idlongrosumr expand_asr6rnumelrAr5masked_scatterrSrLrr6r;r)r,rgrrMrhr6rir\r]rjr~rrkrlrmr[rJspecial_image_maskn_image_tokensn_image_featuresrs r0r>zInternVLModel.forwards(2C1N-TXT_T_TqTq$8$D $++JjJj &1%2J   QU   r1rcr`c |j\}}}}||zdk7s||zdk7r td|j||t||zt||z }|j ddddj }|j|t||zt||zt||dzz }|j ddddj }|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr )rrlrrrrZ)r,rcr`rrrrds r0rbzInternVLModel.pixel_shuffles/>.B.B.D+ E68 L A %)=)Bjk k*.. s6L#893x,?V;W *11!Q1=HHJ*.. F\12C 8L4MsS[_kmn_nSoOp  *11!Q1=HHJr1)NNNNNNNNNNNNNN)r)rDrErF_checkpoint_conversion_mappingrr%r.rZr(rNrrrstrrfrr LongTensorrrrrr rrLr>floatrbrGrHs@r0rRrRgs '=>N%O"~:8+''+$CcN3+), +Z'+*.1537=A59@D8<$(,0/3&*59$(T ##T ''T !. T u//0 T "$u'8'8"9: T   1 12T 'uS$s)^'<=T )1 T D>T $D>T 'tnT d^T !!1!12T \\T -.!T " u11 2#T T l!U\\!!r1rRceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<y) InternVLCausalLMOutputWithPasta Base class for InternVL causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsr6r;rrM)rDrErFrrrr(rNrOrr6rr;rrrMrr1r0rr&s<)-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", torch_dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```NT)rgrrMrhr6rir\r]rjr~rrkrlrmr)rrr)rrr6r;rrMr)rdr~rrpr\r]rrrslicer loss_functionrDrrr6r;rrM)r,rgrrMrhr6rir\r]rrjr~rrkrlrrmr[rr; slice_indicesrrs r0r>z(InternVLForConditionalGeneration.forward~s|2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD.#33!//)) ' ; ;   r1c Nt ||f|||||d|} |ddk(r|| d<| S)N)r6rirMrlrrr)r$prepare_inputs_for_generation) r,rgr6rirrMrlrr[ model_inputsr/s r0rz>InternVLForConditionalGeneration.prepare_inputs_for_generationsVw<  +')))    !  !,8L (r1sequence_length target_lengthr5rc ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer5ror)diagonal)ror3r)rSr(finfominfullrotriuarangerrclonerAr6 masked_fill) rMrrr5rlrr[r_ min_dtype mask_length padding_masks r05_prepare_4d_causal_attention_mask_with_cache_positionzVInternVLForConditionalGeneration._prepare_4d_causal_attention_mask_with_cache_positions<  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r1)NNNNNNNNNNNNNNrN)NNNNNN)'rDrErFr|_tied_weights_keysrr%r.rZr&ModulerrpropertyrSrVrWrrr(r~rNrrrrrr}rrrrrr>r staticmethodr5rrGrHs@r0rrQs"8-"?#, &" ++~ 1/ryy&))''00'+*.1537=A59@D8<-1$(,0/3&*5934.2#n ##n ''n !. n u//0 n "$u'8'8"9: n   1 12n 'uS$s)^'<=n )1 n ))*n D>n $D>n 'tnn d^n !!1!12n c5<</0!n "ell+#n $*+%n & u44 5'n n f<4 444{{ 4  4  44r1r)rr'r4rRr)r)Fcollections.abcr dataclassesrtypingrrrrrr(torch.nnr& activationsr generationr integrationsr modeling_flash_attention_utilsr modeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrrrrrrautorconfiguration_internvlrr get_loggerrDrrr"rrrarcrrrrrrrrrr'r4r@rLrRrrr__all__rr1r0rs.!99 !)7BddF&H   H %Y'JBIIJ(J6% II% <<% % << % U\\* %  %%6NbiiNb KO K KF +E 2!7BII!7L[7ryy[7|  3H I-/"))-/`( BII( V2 72 2 j*o**0"))$ <"9< <@ w+w  wt $<[$< $ m'>m  m` r1