iRddlmZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z m Z dd lmZdd lmZdd lmZmZmZmZdd lmZd dlmZddlmZeedGdde ZeedGdde ZGddej>Z eGddeZ!edGdde!Z"edGd d!e!e Z#gd"Z$y)#) dataclassN)nn)ACT2FN)Cache)GenerationMixin)BaseModelOutputWithPastBaseModelOutputWithPooling ModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)check_model_inputs) AutoModel)VipLlavaConfigzM Base class for VipLlava outputs, with hidden states and attentions.  custom_introc:eZdZUdZdZej dzed<y)VipLlavaModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__x/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/vipllava/modeling_vipllava.pyrr&s 59**T18r$rzT Base class for VipLlava causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) VipLlavaCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrrrr(r r!r"r)r*rr+tupler,rr#r$r%r'r';s &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r$r'c*eZdZdeffd ZdZxZS)VipLlavaMultiModalProjectorconfigcHt|t|jtrdnt |j}t j||jjz|j|_ t j||jjz|jjd|_t|j |_t j|jj|jjd|_y)Nr)epsTbias)super__init__ isinstancevision_feature_layersintlenr LayerNorm vision_config hidden_sizeprojector_layernorm_epsprojector_layernormLinear text_configlinear_1rprojector_hidden_actactlinear_2)selfr0num_feature_layers __class__s r%r6z$VipLlavaMultiModalProjector.__init__Zs ",V-I-I3"OQUXY_YuYuUv#%<< !5!5!A!A AvGeGe$   !5!5!A!A A    * *  &556 &"4"4"@"@&BTBTB`B`gkl r$c|j|}|j|}|j|}|j|}|SN)r?rBrDrE)rFr+s r%forwardz#VipLlavaMultiModalProjector.forwardisB00?  m4 /  m4 r$)rrrrr6rK __classcell__rHs@r%r/r/Ys m~ mr$r/c<eZdZUeed<dZdZdZdZdZ dZ dZ dZ dZ y)VipLlavaPreTrainedModelr0model)imagetextTr*N)rrrrr"base_model_prefixinput_modalitiessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_flex_attn_supports_attention_backendr#r$r%rOrOqs= (&*#"3N!"&r$rOzx The VipLlava model which consists of a vision backbone and a language model, without a language modeling head. c6eZdZddiZdeffd ZdZdZee d dd e jd e e e zd zd ed zd eedeezf dZde j(de jde jfdZe dde j(d zd e jd zde j,d zde j(d zded zde jd zd e e e zd zded zded zd ed zded zde j(d zdeezfdZxZS) VipLlavaModel^language_model.modellanguage_modelr0ct||tj|j|_t ||_tj|j|_ |jyrJ) r5r6r from_configr< vision_towerr/multi_modal_projectorrAr_ post_initrFr0rHs r%r6zVipLlavaModel.__init__sY  %11&2F2FG%@%H"'33F4F4FG r$c6|jjSrJ)r_get_input_embeddingsrFs r%rgz"VipLlavaModel.get_input_embeddingss""7799r$c:|jj|yrJ)r_set_input_embeddingsrFvalues r%rjz"VipLlavaModel.set_input_embeddingss 007r$zWObtains image last hidden states from the vision tower and apply multimodal projection.rN pixel_valuesr8output_hidden_stateskwargsreturnc j||n|jj}|j|fddd|}t|tr|j |ddddf}n<|Dcgc]}|j |ddddf}}t j|d}|j|}||_ |Scc}w)\ pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`): The tensors corresponding to the input images. vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. NT)rn return_dictr)dim) r0r8rbr7r9r+r catrc pooler_output)rFrmr8rnro image_outputsimage_featuresindexs r%get_image_featuresz VipLlavaModel.get_image_featuress&&;%F !DKKLmLm *))  !%    +S 1*889NOPQSTSUPUVNVkkEm99%@ABGkNk"YY~2>N33NC&4 # lsB0 input_ids inputs_embedsrycN|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd|jdz}|jdj|j|j }t||j|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )dtypedevicertrrz6Image features and image tokens do not match, tokens: z , features: )rgr tensorr0image_token_idlongrallsumshape unsqueeze expand_astornumel)rFr|r}ryspecial_image_maskn_image_tokensn_image_featuress r%get_placeholder_maskz"VipLlavaModel.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r$attention_mask position_idsr* use_cacheoutput_attentionsrscache_positionc | | n|jj} | | n|jj} | | n|jj} ||n|jj}|du|duz r t d||j |}|j|j||dj}|j|j|j}|j|||}|j||}|jd|||||| | d| d | }t|j |j"|j$|j&|nd}| r|S|j)S) z vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Nz:You must specify exactly one of input_ids or inputs_embedsT)rmr8rs)r}ry) rrr*r}rrrnrsr)last_hidden_stater*r+r,rr#)r0rrnuse_return_dictr8 ValueErrorrgr{rwrrrrmasked_scatterr_rrr*r+r,to_tuple)rFr|rmrrr*r}r8rrrnrsr lm_kwargsryroutputsoutputs r%rKzVipLlavaModel.forwards,2C1N-TXT_T_TqTq$8$D $++JjJj &1%2JPT  %v;&//*;;r$)NN) NNNNNNNNNNNN)rrr_checkpoint_conversion_mappingrr6rgrjrrr r!r9listboolr rr-r r{ LongTensorrTensorrrrKrLrMs@r%r]r]s !"2&"~:8n 9=,0 #''# #T#Y5##Tk # +, # + + ##J"))":?:K:K"]b]n]n"0.215.204(,268>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, VipLlavaForConditionalGeneration >>> model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", device_map="auto", dtype=torch.float16) >>> processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf") >>> prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: \n{}###Assistant:" >>> question = "Can you please describe this image?" >>> prompt = prompt.format(question) >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16) >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=20) >>> processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True) The image features a brown and white cat sitting on a green surface, with a red ball in its ```NT) r|rmrrr*r}rr8rrnrsrr)r)rr)r(r)r*r+r,rr#)r0rrnrr8rPr7r9slicer loss_functionrArr'r*r+r,r)rFr|rmrrr*r}r8rrrrnrsrrrrr+ slice_indicesr)r(s r%rKz(VipLlavaForConditionalGeneration.forwardOs[p2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %%VFt{{OfOfOqOq%rD-#33!//)) ' ; ;   r$c ht ||f||||||d| } |s| jdds|| d<| S)N)r*r}rrris_first_iterationrTrm)r5prepare_inputs_for_generationget) rFr|r*r}rmrrrrro model_inputsrHs r%rz>VipLlavaForConditionalGeneration.prepare_inputs_for_generations\w<   +')))1     VZZ T%B ,8L (r$rJ)NNNNNNNNNNNNNr)NNNNNNF)rrrr_tied_weights_keysrr6rgrjrModulerrr r!r9rr rr-r r{rrrrrr'rKrrLrMs@r%rr s:#9.#@$- &" +,VW~ 1/ryy9= ''  #T#Y5 +,  + +   "u5.215.204(,268<*.!%)-,0#'26-._ ##d*_ ''$._  t+ _ &&- _  _ ((4/_  #T#Y5_   4'_ $;_  $;_ #Tk_ D[_ ((4/_ ell*_ " / /#_ 6_ H   r$r)r]rrO)% dataclassesrr r activationsr cache_utilsr generationrmodeling_outputsr r r modeling_utilsr processing_utilsr utilsrrrr utils.genericrautorconfiguration_vipllavarrr'rr/rOr]r__all__r#r$r%rs*" ! )``-&aa/2  9"9 9  9  9[9  90"))0 'o ' ' W<+W<  Wm  m` [r$