UhbbddlmZddlmZmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZdd lmZdd lmZmZmZd d lmZd dlmZeGddeZeGddeZGddej8ZeGddeZedGddeZedGddee Z gdZ!y)) dataclass)ListOptionalTupleUnionN)nn)ACT2FN)GenerationMixin)BaseModelOutputWithPast ModelOutput)PreTrainedModel)auto_docstringcan_return_tupleis_torchdynamo_compiling) AutoModel)VipLlavaConfigc:eZdZUdZdZeejed<y)VipLlavaModelOutputWithPasta Base class for VipLlava outputs, with hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/vipllava/modeling_vipllava.pyrr%s88<%"3"34;r!rceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<y) VipLlavaCausalLMOutputWithPasta Base class for VipLlava causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrrrr%rrrrr&r'rr(rr)rr r!r"r$r$Fs<)-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8r@rA)rBr(s r"forwardz#VipLlavaMultiModalProjector.forward~sB00?  m4 /  m4 r!)rrrrr2rG __classcell__rDs@r"r+r+ns m~ mr!r+c:eZdZeZdZdZdZdZdZ dZ dZ dZ dZ dZy)VipLlavaPreTrainedModelTr'c$t|jd|jjj}t |t j rY|jjjd||j%|jjjyyt |t jrJ|jjjd|jjjyy)Ninitializer_rangeg)meanstdg?)getattrr,get_text_configrNr3rr<weightdatanormal_r0zero_r7fill_)rBmodulerPs r" _init_weightsz%VipLlavaPreTrainedModel._init_weightssdkk#6 8S8S8U8g8gh fbii ( MM   & &CS & 9{{&   &&('  - MM   $ $S ) KK   " " $.r!N)rrrr config_classbase_model_prefixsupports_gradient_checkpointing_skip_keys_device_placement_supports_cache_class_supports_flash_attn_2_supports_sdpa_supports_quantized_cache_supports_static_cache_supports_attention_backendrYr r!r"rKrKsA!L&*#"3 !N $!"& %r!rKzx The VipLlava model which consists of a vision backbone and a language model, without a language modeling head. ) custom_introceZdZddiZdeffd ZdZdZdejde e e e ffd Z e dd ejdejd eej"d eejd ee ejdeejdee e e e fdeedeedeedeedeejde eeffdZxZS) VipLlavaModelzlanguage_model.modellanguage_modelr,ct||tj|j|_t ||_tj|j|_ |jyrF) r1r2r from_configr8 vision_towerr+multi_modal_projectorr=rg post_initrBr,rDs r"r2zVipLlavaModel.__init__sY  %11&2F2FG%@%H"'33F4F4FG r!c6|jjSrF)rgget_input_embeddingsrBs r"roz"VipLlavaModel.get_input_embeddingss""7799r!c:|jj|yrF)rgset_input_embeddingsrBvalues r"rrz"VipLlavaModel.set_input_embeddingss 007r! pixel_valuesr4c"|j|d}t|tr|j|ddddf}n<|Dcgc]}|j|ddddf}}t j |d}|j |}|Scc}w)aW Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layers (`Union[int, List[int]]`): The vision feature layer, or the list of indexes of the layers to select the vision feature. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). T)output_hidden_statesNr)dim)rjr3r5r(rcatrk)rBrur4 image_outputsimage_featuresindexs r"get_image_featuresz VipLlavaModel.get_image_featuress)),T)R  +S 1*889NOPQSTSUPUVNVkkEm99%@ABGkNk"YY~2>N33NClsB  input_idsattention_mask position_idsr' inputs_embeds use_cacheoutput_attentionsrw return_dictcache_positionreturnc j| | n|jj} | | n|jj} | | n|jj} ||n|jj}|du|duz r t d| | t d||j |}|"|j||}||jjk(jd}|j|j|j}ts{||j|jk7rW||jjk(j}|j d|j dz}t d|d ||j|j|j"}|j%||}|j&d |||||| | d | d | }t)|j*|j,|j.|j0|nd }| r|S|j3S)z vision_feature_layers (`Union[int, List[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Nz:You must specify exactly one of input_ids or inputs_embedszdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either one)rur4rxrrz6Image features and image tokens do not match: tokens: z , features T) rrr'rrrrwrr)last_hidden_stater'r(r)rr )r,rrwuse_return_dictr4 ValueErrorror~image_token_id unsqueeze expand_astodevicernumelsumshapedtypemasked_scatterrgrrr'r(r)to_tuple)rBrrurrr'rr4rrrwrr lm_kwargsr|special_image_maskn_image_tokensn_image_featuresoutputsoutputs r"rGzVipLlavaModel.forwardsq,2C1N-TXT_T_TqTq$8$D $++JjJj &1%2JPT  %v;&//*;;r!) NNNNNNNNNNNN)rrr_checkpoint_conversion_mappingrr2rorrrrrr5rr~r LongTensorrTensorboolrrrGrHrIs@r"rfrfs '=>N%O"~:8u/@/@Y^_bdhildm_mYn4'+*.1537=A59AE$(,0/3&*59M<##M<''M<!. M< u//0 M< "$u'8'8"9: M<  1 12M< (c49n(=>M<D>M<$D>M<'tnM<d^M<!!1!12M< u11 2M<M>> import torch >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, VipLlavaForConditionalGeneration >>> model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", device_map="auto", torch_dtype=torch.float16) >>> processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf") >>> prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: \n{}###Assistant:" >>> question = "Can you please describe this image?" >>> prompt = prompt.format(question) >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16) >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=20) >>> processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True) The image features a brown and white cat sitting on a green surface, with a red ball in its ```NT) rrurrr'rrr4rrwrrr)r&rr)r%r&r'r(r)rr )r,rrwrr4rr3r5slicer loss_functionr=rr$r'r(r)r)rBrrurrr'rr4rrrrwrrrrrr( slice_indicesr&r%s r"rGz(VipLlavaForConditionalGeneration.forwardPs[l2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %%VFt{{OfOfOqOq%rD-#33!//)) ' ; ;   r!c Nt ||f|||||d|} |ddk(r|| d<| S)N)r'rrrrrru)r1prepare_inputs_for_generation) rBrr'rrurrrkwargs model_inputsrDs r"rz>VipLlavaForConditionalGeneration.prepare_inputs_for_generationsVw<  +')))    !  !,8L (r!sequence_length target_lengthr batch_sizec ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerrr)diagonal)rrxr)ryrfinfominfullrtriuarangereshapeexpandclonerr masked_fill) rrrrrrr causal_mask min_dtype mask_length padding_masks r"5_prepare_4d_causal_attention_mask_with_cache_positionzVVipLlavaForConditionalGeneration._prepare_4d_causal_attention_mask_with_cache_positions<  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r!)NNNNNNNNNNNNNr)NNNNNN)$rrrr_tied_weights_keysrr2rorrrModulerrpropertyrgrjrkrrrrrrrrrr5rrr$rGr staticmethodrrrHrIs@r"rr#sv"8-"?#, &" ++~ 1/ryy&))''00'+*.1537=A59AE-1$(,0/3&*5934] ##] ''] !. ] u//0 ] "$u'8'8"9: ]   1 12]  (c49n(=>] ))*] D>] $D>] 'tn] d^] !!1!12] c5<</0] " u44 5#] ] D<4 444{{ 4  4  44r!r)rfrrK)" dataclassesrtypingrrrrrr activationsr generationr modeling_outputsr r modeling_utilsrutilsrrrautorconfiguration_vipllavarrr$rr+rKrfr__all__r r!r"rs,"// !)D-OO2 <"9< <@ $<[$< $\  \~ [r!