iIdZddlmZddlZddlmZddlmZddlmZddl m Z dd l m Z m Z mZdd lmZdd lmZdd lmZmZmZmZdd lmZddlmZddlmZej<eZ eedGdde Z!eedGddeZ"GddejFZ$eGddeZ%edGdde%Z&ed Gd!d"e%e Z'gd#Z(y)$zPyTorch Llava model.) dataclassN)nn)ACT2FN)Cache)GenerationMixin)BaseModelOutputWithPastBaseModelOutputWithPooling ModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringloggingtorch_compilable_check)check_model_inputs) AutoModel) LlavaConfigzJ Base class for Llava outputs, with hidden states and attentions.  custom_introc:eZdZUdZdZej dzed<y)LlavaModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__r/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/llava/modeling_llava.pyrr$s 59**T18r$rzQ Base class for Llava causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) LlavaCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rrrrr(r r!r"r)r*rr+tupler,rr#r$r%r'r'9s &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r$r'c*eZdZdeffd ZdZxZS)LlavaMultiModalProjectorconfigct|t|jtrdnt |j}t j|jj|z|jj|j|_ t|j|_t j|jj|jj|j|_y)Nrbias)super__init__ isinstancevision_feature_layerintlenrLinear vision_config hidden_size text_configmultimodal_projector_biaslinear_1rprojector_hidden_actactlinear_2)selfr0num_feature_layers __class__s r%r5z!LlavaMultiModalProjector.__init__Xs ",V-H-H#"NQTWX^XsXsTt  , ,/A A    * *11  &556    * *F,>,>,J,JQWQqQq  r$cl|j|}|j|}|j|}|SN)r?rArB)rCimage_featuresr+s r%forwardz LlavaMultiModalProjector.forwardfs2 n5 /  m4 r$)rrrrr5rI __classcell__rEs@r%r/r/Ws  {  r$r/c<eZdZUeed<dZdZdZdZdZ dZ dZ dZ dZ y)LlavaPreTrainedModelr0model)imagetextTr*N)rrrrr"base_model_prefixinput_modalitiessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_flex_attn_supports_attention_backendr#r$r%rMrMms= (&*#"3N!"&r$rMzu The Llava model which consists of a vision backbone and a language model, without a language modeling head. cneZdZddiZdeffd ZdZdZede d  dd e jd e e e zd zded zded zdeedeezf dZde j*de jde jfdZede dde j*d zd e jd zde j.d zde j*d zded zde jd zd e e e zd zded zde j*d zde j.d zdeedeezfdZxZS) LlavaModel^language_model.modellanguage_modelr0ct||tj|j|_t ||_tj|j|_ |jyrG) r4r5r from_configr; vision_towerr/multi_modal_projectorr=r] post_initrCr0rEs r%r5zLlavaModel.__init__sY  %11&2F2FG%=f%E"'33F4F4FG r$c6|jjSrG)r]get_input_embeddingsrCs r%rezLlavaModel.get_input_embeddingss""7799r$c:|jj|yrG)r]set_input_embeddingsrCvalues r%rhzLlavaModel.set_input_embeddingss 007r$Ftie_last_hidden_stateszWObtains image last hidden states from the vision tower and apply multimodal projection.rN pixel_valuesr7vision_feature_select_strategyoutput_hidden_stateskwargsreturnc |jDcic] \}}| || }}}|j|fddd|}t|tr |j|} |dk(r\| ddddf} nP|D cgc]} |j| } } |dk(r| D cgc] } | ddddf} } t j | d} |j| } |jdt j|d| j|jjzjdj}t j| jd |} n t!| } | |_|Scc}}wcc} wcc} w) NT)ro return_dictdefaultrdim image_sizes)devicer)itemsr`r6r8r+r catraget as_tensorry patch_sizeprodtolistsplitsqueezelist pooler_output)rCrmr7rnrorpkv image_outputsselected_image_feature layer_idxhs_poolhsrH split_sizess r%get_image_featureszLlavaModel.get_image_featuress$*<<>C41aQ]!Q$CC)))  !%    *C 0%2%@%@AU%V "-:)?12)F&Ocd)}229=dGd-:/672ae977%*YYwB%? "334JK ::m $ 0 !6~?T?TUY]YjYjYuYuu"  #[[)?)?)BKPN!.1N&4 #KD e8s E2E21E8E= input_ids inputs_embedsrHcN|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd|jdz}|jdj|j|j }t||j|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )dtyperyrurrz6Image features and image tokens do not match, tokens: z , features: )rer tensorr0image_token_idlongryallsumshape unsqueeze expand_astornumel)rCrrrHspecial_image_maskn_image_tokensn_image_featuress r%get_placeholder_maskzLlavaModel.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r$attention_mask position_idsr*cache_positionrxc |du|duz r td||j|}||j|||| dj} t j | dj |j|j} |j||| } |j| | }|jd ||||| d| }t|j|j|j|j | SdS) Nz:You must specify exactly one of input_ids or inputs_embedsT)rmr7rnrxrsrrv)rrH)rrr*rr)last_hidden_stater*r+r,rr#) ValueErrorrerrr r{rryrrmasked_scatterr]rrr*r+r,)rCrrmrrr*rr7rnrrxrprHroutputss r%rIzLlavaModel.forwardsK  -t"; <YZ Z  7D557 BM  #!44)%9/M' 5 m #YY~1=@@AUAUWdWjWjkN!%!:!:~";" *889K^\M%$%% )%+')    (%77#33!//))2>2J   QU   r$)NNN) NNNNNNNNNN)rrr_checkpoint_conversion_mappingrr5rerhrrr r!r8rstrboolr rr-r r LongTensorrTensorrrrIrJrKs@r%r[r[}s !"2&"{:8u5n 8<59,0 -''-"DIo4-),d - #Tk - +, - + +-6-^"))":?:K:K"]b]n]n"0u5.215.204(,267;5926+/1 ##d*1 ''$.1  t+ 1 &&- 1  1 ((4/1 "DIo41 ),d 1 ((4/1 \\D(1 +,1  ) )1 61 r$r[zS The LLAVA model which consists of a vision backbone and a language model. c teZdZdddddZddiZdeffd Zd Zd Zd e jfd Z e d!de jdeeezdzdedzdeed eezf dZede d"de j0dzde jdzde j2dzde j0dzdedzde jdzdeeezdzdedzde j0dzde j0dzdee j2zde j2dzdeed eezfdZ d#fd ZxZS)$LlavaForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)r\z ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightz(model.language_model.embed_tokens.weightr0ct||t||_t j |j j|j jd|_ |jy)NFr2) r4r5r[rNrr:r=r< vocab_sizerrbrcs r%r5z&LlavaForConditionalGeneration.__init__$sS  ' yy!3!3!?!?ASASA^A^ejk  r$c6|jjSrG)rNrerfs r%rez2LlavaForConditionalGeneration.get_input_embeddings*szz..00r$c:|jj|yrG)rNrhris r%rhz2LlavaForConditionalGeneration.set_input_embeddings-s ''.r$rqc|jSrG)rrfs r%get_output_embeddingsz3LlavaForConditionalGeneration.get_output_embeddings0s ||r$Nrmr7rnrpc B|jjd|||d|S)N)rmr7rnr#)rNr)rCrmr7rnrps r%rz0LlavaForConditionalGeneration.get_image_features3s5-tzz,, %!5+I    r$Frkrrrr*rlabelsrlogits_to_keeprxc  |jd||||||||| | d | }|d}t| tr t| dn| }|j |dd|ddf}d}| 4|j d|| |j jjd| }t|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, LlavaForConditionalGeneration >>> model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> prompt = "USER: \nWhat's the content of the image? ASSISTANT:" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "USER: \nWhat's the content of the image? ASSISTANT: The image features a busy city street with a stop sign prominently displayed" ```) rrmrrr*rr7rnrrxrN)r)rr)r(r)r*r+r,rr#)rNr6r8slicer loss_functionr0r=rr'r*r+r,r)rCrrmrrr*rr7rnrrrrxrprr+ slice_indicesr)r(s r%rIz%LlavaForConditionalGeneration.forwardBs^$**  %)%+'!5+I)#      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD+#33!//)) ' ; ;   r$c ht ||f||||||d| } |s| jdds|| d<| S)N)r*rrrris_first_iteration use_cacheTrm)r4prepare_inputs_for_generationr|) rCrr*rrmrrrrrp model_inputsrEs r%rz;LlavaForConditionalGeneration.prepare_inputs_for_generations\w<   +')))1     VZZ T%B ,8L (r$)NN) NNNNNNNNNNrN)NNNNNNF)rrrr_tied_weights_keysrr5rerhrModulerrr r!r8rrr rr-r rrrrrr'rIrrJrKs@r%rrs=#9.#@$- &" +,VW{ 1/ryy8<59  ''  "DIo4  ),d  +,  + +    u5.215.204(,267;59*.26-.+/M ##d*M ''$.M  t+ M &&- M  M ((4/M "DIo4M ),d M   4'M ((4/M ell*M \\D(M +,M  , ,M 6M d   r$r)rrMr[))r dataclassesrr r activationsr cache_utilsr generationrmodeling_outputsr r r modeling_utilsr processing_utilsr utilsrrrr utils.genericrautorconfiguration_llavar get_loggerrloggerrr'rr/rMr[r__all__r#r$r%rs/! ! )``-&XX/,   H %  96 9  9  9+9  90ryy, '? ' ' Q %Q  Q h X$8/X  Xv Rr$