iiBdZddlmZddlmZddlZddlmZddlmZddl m Z dd l m Z dd l mZdd lmZdd lmZmZdd lmZddlmZddlmZmZmZmZmZmZddlm Z ddl!m"Z"ejFe$Z%eedGddeZ&eedGddeZ'GddejPZ)dejTdzdejTdzdedzfd Z+ d3d!e d"ejTd#ejTdzd$ejTd%edzd&ejTdzdejTdzd'ejXdzd(e-dzd)e-dzde.fd*Z/eGd+d,eZ0ed-Gd.d/e0Z1ed-Gd0d1e0e Z2gd2Z3y)4zPyTorch PaliGemmamodel.)Callable) dataclassN)nn)Cache)PreTrainedConfig)GenerationMixin)create_masks_for_generate)FlashAttentionKwargs)BaseModelOutputWithPastBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions.  custom_introc:eZdZUdZdZej dzed<y)PaligemmaModelOutputWithPasta image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__z/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/paligemma/modeling_paligemma.pyrr-s 59**T18r'rzU Base class for PaliGemma causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) PaliGemmaCausalLMOutputWithPasta8 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)rr r!r"r+r#r$r%r,r-rr.tupler/rr&r'r(r*r*=s &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r'r*c*eZdZdeffd ZdZxZS)PaliGemmaMultiModalProjectorconfigct|tj|jj |jj d|_y)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr3 __class__s r(r8z%PaliGemmaMultiModalProjector.__init__\s; ii 4 4 @ @&BVBVBeBelpq r'c(|j|}|SN)r=)r?image_featuresr.s r(forwardz$PaliGemmaMultiModalProjector.forward`s N3 r')rr r!rr8rD __classcell__r@s@r(r2r2[srrr'r2token_type_idsimage_group_idsreturnc Zydtdtdtdtdtf fd }|S)z This function adds the correct offsets to the `q_idx` and `kv_idx` as the torch API can only accept lengths, not start and end indices. N batch_idxhead_idxq_idxkv_idxrIc:tj| jdk|d}tj| jdk|d} ||f}tj| jdk|d} ||f}tj| jdk|d} ||f}tj| jdk|d} ||f} tj| jdk| d} |dk(|dk(z} || k(} | | zS)Nrr)r#whereshape)rKrLrMrN safe_q_idx safe_kv_idxtoken_type_ids_at_q_idxtoken_type_ids_at_kv_idximage_group_ids_at_q_idximage_group_ids_at_kv_idxis_image_blocksame_image_blockrHrGs r( inner_maskz0token_type_ids_mask_function..inner_maskrsM[[)=)=a)@!@%K kk&>+?+?+B"BFAN "0J1F"G"'++en6J6J16M.MOfhi"j#1)[2H#I #(;;v8L8LQ8O/OQikl#m #29j3H#I #(;;u7L7LQ7O/OQikm#n $3I{4J$K!$)KK9N9Nq9Q0QSlnp$q!1Q6;SWX;XY37PP 000r')intbool)rGrHr[s`` r(token_type_ids_mask_functionr^fs>1c1S11c1d12 r'r3 input_embedsattention_maskcache_positionr- position_ids pixel_values is_trainingis_first_iterationc  |r | td|j|||||d} | r| n|duxs|j xs|du} | s| jdds<|d|z }n4tj dt j|dddddf}|| r|dk(j|j} tjj| d d dddd f} | | z}t j|jd dz }t j| |t j |d }t#|j|j|| d <t%di| S)a" Overwrites the base `create_masks_for_generate` with `token_type_ids` masking to create the causal mask mapping for all kinds of forward passes. Paligemma uses a bidirectional mask on the prompt tokens. Uses `pixel_values` as an optional input to disambiguate edge cases. Nz;`token_type_ids` is required as a model input when training)r3r_r`rar-rb use_cacheTrzIt is a prefill stage but The `token_type_ids` is not provided. We recommend passing `token_type_ids` to the model to prevent bad attention masking.r)rr)valuerP)dimor_mask_functionr&) ValueErrorget_text_configis_initializedgetlogger warning_oncer# ones_liketodevicer functionalpadcumsumr\rQ full_liker^r )r3r_r`rar-rbrGrcrdrekwargs mask_kwargsis_imageis_previous_imagenew_image_startrHs r(create_causal_mask_mappingr}s&~-VWW((*$((*$ K  %g_-K-K)Kg|cgOg K!>  %/N   Z  #__\:1a7CN !&8 #a'++N,A,ABMM--ha-HCRCP"&7%77,,':':'!D!D!R  r'c6|jjSrB)rget_input_embeddingsr?s r(rz#PaliGemmaModel.get_input_embeddingss""7799r'c:|jj|yrB)rset_input_embeddingsr?rhs r(rz#PaliGemmaModel.set_input_embeddingss 007r'zWObtains image last hidden states from the vision tower and apply multimodal projection.rrcrxrIc |j|fddi|}|j}|j|}||jjj dzz }||_|S)N return_dictTg?)rlast_hidden_staterr3rr; pooler_output)r?rcrx image_outputsselected_image_featurerCs r(get_image_featuresz!PaliGemmaModel.get_image_featuressk*)),SDSFS !.!@!@334JK'4;;+B+B+N+NPS+ST&4 #r' input_ids inputs_embedsrCcN|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd|jdz}|jdj|j|j }t||j|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )rrsrPrrz6Image features and image tokens do not match, tokens: z , features: )rr#tensorr3image_token_idlongrsallsumrR unsqueeze expand_asrrrnumel)r?rrrCspecial_image_maskn_image_tokensn_image_featuress r(get_placeholder_maskz#PaliGemmaModel.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r'Nr`rbr-rGralabelsrgoutput_attentionsoutput_hidden_statesrc b|du|duz r td| | n|jj} | | n|jj} | | n|jj} |R|jj |j k\r/||jj k(}|j}d||<n|}||j|}|F||jnd}tj|||jdz|j}||jddz}|i|j|dj }|j#|j|j$}|j'|||}|j)||}t+|x}t,s(t/|j||||||||j0 }|j2d ||||| | | d|d |}t5|j6|j8|j:|j<| Sd S)  labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrr)rsT)r)rrC)rd) r`rbr-rrgrrrra)rr-r.r/rr&)rkr3rruse_return_dictrrclonerget_seq_lengthr#arangerRrsrrrrrrrmasked_scatter isinstancedictr}trainingrrrr-r.r/)r?rrcr`rbr-rGrarrrgrrrrxr llm_input_idspast_seen_tokensrCcausal_mask_mappingoutputss r(rDzPaliGemmaModel.forward,s}b  -t"; <YZ Z1B1N-TXT_T_TqTq$8$D $++JjJj &1%2J   QU   r') NNNNNNNNNNNNN)rr r!_checkpoint_conversion_mappingaccepts_loss_kwargsrr8rrrrr#r$rrr0r r LongTensorrTensorrr]r rrDrErFs@r(rrs  '=>N%O"  :8n !-- 9?@R9S  + +  "))":?:K:K"]b]n]n"0.215.204(,262626*.!%)-,0#'v ##d*v ''$.v  t+ v &&- v  v ((4/v ((4/v ((4/v   4'v $;v  $;v #Tkv D[v -.v - -!v v r'rc$eZdZdddddZddiZdeffd Zd Zd Ze d e jd e e fdZee d#de j dzd e jdzde j"dzde j dzdedzde j dzde j dzde jdzde j dzdedzdedzdedzdedzdee j"zd e e deezf dZ d$fd Ze d%ded e j"de j"dzde j"dedzde j"dzde j"dzd!edzdefd"ZxZS)&!PaliGemmaForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)z^language_model.modelz ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightz(model.language_model.embed_tokens.weightr3ct||t||_t j |j j|j jd|_ |jy)NFr5) r7r8rrrr9rr;rrrr>s r(r8z*PaliGemmaForConditionalGeneration.__init__sS  #F+ yy!3!3!?!?ASASA^A^ejk  r'c6|jjSrB)rrrs r(rz6PaliGemmaForConditionalGeneration.get_input_embeddingsszz..00r'c:|jj|yrB)rrrs r(rz6PaliGemmaForConditionalGeneration.set_input_embeddingss ''.r'rcrxc <|jj|fi|SrB)rr)r?rcrxs r(rz4PaliGemmaForConditionalGeneration.get_image_featuress,tzz,,\DVDDr'Nrr`rbr-rGrarrrgrrrlogits_to_keeprIc >| | n|jj} | | n|jj} | | n|jj} |jd|||||||| | | | d|d |}|d}t |t r t| dn|}|j|dd|ddf}d}| 4|jd|| |jjjd|}t|||j|j|j|j S)rNT) rrcrGr`rbr-rrgrrrrrar)r,rr)r+r,r-r.r/rr&)r3rrrrrr\slicer loss_functionrrr*r-r.r/r)r?rrcr`rbr-rGrarrrgrrrrrxrr. slice_indicesr,r+s r(rDz)PaliGemmaForConditionalGeneration.forwardsSb2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD/#33!//)) ' ; ;   r'c t||f|||||| | || d | }|jd |dxxdz cc<| s| s||d<|S)N) r-rr`rbrargrrGrerbrrc)r7prepare_inputs_for_generationrn)r?rr-rrarbrcr`rGrgrrrerx model_inputsr@s r(rz?PaliGemmaForConditionalGeneration.prepare_inputs_for_generation!s{"w<   +')%)))1       N + 7  (A - ( Y+7L (r'r_rec t|||||||fd|i|jD cic]\} } | dk7s | | c} } Scc} } w)Nrerc)r}items) r3r_r`rar-rbrGrerxkvs r(r z;PaliGemmaForConditionalGeneration.create_masks_for_generateMs`*          2  !' F1!~2Eq!tF  Gs ==)NNNNNNNNNNNNNr) NNNNNNNTNNF)NF)rr r!r_tied_weights_keysrr8rrrr#r$rrrrrrrr]r\r0r*rDr staticmethodrrr rErFs@r(rrs"8-"?#, &" +,VW 1/Eu/@/@EFSeLfEE.215.204(,262626*.!%)-,0#'-.X ##d*X ''$.X  t+ X &&- X  X ((4/X ((4/X ((4/X   4'X $;X  $;X #TkX D[X ell*X +,!X " 0 0#X X z *X/3*/   ll  t+     llT)   t+ !4K    r'r)rrr)NNFN)4r"collections.abcr dataclassesrr#r cache_utilsrconfiguration_utilsr generationr masking_utilsr modeling_flash_attention_utilsr modeling_outputsr r modeling_utilsrprocessing_utilsrutilsrrrrrrautorconfiguration_paligemmar get_loggerrrorr*Moduler2rr^r$r]rr}rrr__all__r&r'r(rs2$!  3)6BS-&4   H %  9#:9  9  9k9  90299%LL4'%\\D(%_%^+/-1$&*G4 G4,,G4LL4'G4LL G4 T\ G4 ,,% G4LL4'G4##d*G4G4t G4 G4T ' ' ' x -x  x v x (@/x  x v ^r'