iߔdddlZddlmZddlmZddlZddlmZddlm Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*edGddejVZ, dEdejVdejZdejZdejZdejZdzde.de.fdZ/Gdd ejVZ0ee!d!"Gd#d$eZ1Gd%d&ejVZ2Gd'd(ejVZ3Gd)d*ejVZ4ejje,d+Z6Gd,d-eZ7Gd.d/ejVZ8e!Gd0d1eZ9e!Gd2d3e9Z:e!Gd4d5eZ;Gd6d7ejVZee!d>"Gd?d@eZ?e!dA"GdBdCe;eZ@gdDZAy)FN)Callable) dataclass)initialization)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringtorch_compilable_check torch_int)check_model_inputs) AutoModel)InternVLConfigInternVLVisionConfigRMSNormc,eZdZdfd ZdZdZxZS)InternVLVisionRMSNormct|tjt j ||_||_y)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s x/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/internvl/modeling_internvl.pyr"zInternVLVisionRMSNorm.__init__.s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetor%float32powmeanrsqrtr(r')r) hidden_states input_dtypevariances r-forwardzInternVLVisionRMSNorm.forward6sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r.c^t|jjd|jS)Nz, eps=)tupler'shaper(r)s r- extra_reprz InternVLVisionRMSNorm.extra_repr=s* ))*+6$2G2G1HIIr.)gư>)__name__ __module__ __qualname__r"r;r@ __classcell__r,s@r-rr,s$;Jr.rmodulequerykeyvalueattention_maskscalingdropoutc |}|} tj||jdd|z} |#|ddddddd|jdf} | | z} tj j | d} tj j| ||j} tj| | } | jddj} | | fS)Nrrr0dim)ptrainingr) r%matmul transposer>r# functionalsoftmaxrLrR contiguous) rFrGrHrIrJrKrLkwargs key_states value_states attn_weights causal_mask attn_outputs r-eager_attention_forwardr^AsJL<<z';';Aq'ABWLL!$Q1.D 0@0@0D.D%DE #k1 ==((2(>L==((6??([L,,|\:K''1-88:K  $$r.cteZdZdZdeffd Z d dejdejdzdee fdZ xZ S) InternVLVisionAttentionz+Attention Class for InternVL Vision Encoderconfigct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ |j}|j}d|_ tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j|_|dkDrtj*|ntj,|_ |rt/|jntj,|_|rt/|j|_ytj,|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r!r"rar* embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr#Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityrq_normk_norm)r)ra proj_dropoutqk_normr,s r-r"z InternVLVisionAttention.__init___s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!900 $$ii0NU[UjUjk ii0NU[UjUjk ii0NU[UjUjk " $..$.. I>JQ>N"**\":TVT_T_Ta?F+DNN;BKKM ?F+DNN; BKKM r.Nr8rJrXc r|j\}}}|j|}|j|}|j|} |j |}|j |}|j |||j|jjdd}|j |||j|jjdd}| j|||j|jjdd} tj|jjt} | |||| |f|j sdn |j"|j$dd|\} } | j |||j&} |j)| } |j+| } | | fS)NrrF)rLrKrn)sizerqrrrsrwrxreshapergrhrTviewr get_interfacera_attn_implementationr^rRrkrjrertrl)r)r8rJrX batch_sizeseq_len_ query_statesrYrZattention_interfacer]r[outputs r-r;zInternVLVisionAttention.forward{s "/!3!3!5 GQ{{=1 [[/ {{=1 {{<0 [[, #++JQUQ^Q^_iijkmno '' GT^^T]][eefgijk #((Wdnndmm\ffghjkl (?(M(M KK , ,.E) %8      % $}}C$2H2HJJ %  % ! \"))*gt~~N &&{3((0|##r.N) rArBrC__doc__rr"r%Tensorrrr;rDrEs@r-r`r`\sN5Z3Z>/3'$||'$ t+'$+, '$r.r`z7 Class for outputs of [`InternVLVisionModel`].  custom_introceZdZdZy)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)rArBrCrr.r-rrsr.rcZeZdZdZfdZdej dej fdZxZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c^t||j|j}}|j|j }}|d|dz|d|dzz}|d|dz|d|dzf}||_||_||_||_||_tj|||||_ y)Nrr) kernel_sizestride) r!r" image_size patch_size num_channelsr* num_patches patch_shaper#Conv2d projection) r)rarrrr*rrr,s r-r"z&InternVLVisionPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk !!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&))L+:^hir. pixel_valuesreturnc|j\}}}}||jk7r td|j|j |jj j }|jdjdd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rr) r>rrirr3r'r2flattenrT)r)rrrheightwidth embeddingss r-r;z%InternVLVisionPatchEmbeddings.forwards2>2D2D/ L&% 4,, ,w __\__T__5K5K5Q5Q%RS ''*44Q: r.) rArBrCrr"r%rr;rDrEs@r-rrs) j ELL U\\ r.rceZdZdZdeddffd Zdejdededejfd Z d d ejd ejdzdejfd Z xZ S)InternVLVisionEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rarNc2t|tjt j dd|j |_|jr:tjt j dd|j |_ nd|_ t||_ |j|_ t|jtj j"r |jn|j|jf|_|jj$}|j&r=tjt j d|dz|j |_nd|_tj*|j,|_y)Nr)r!r"r#r$r%zerosr* cls_tokenuse_mask_token mask_tokenrpatch_embeddingsr isinstancer collectionsabcIterabler use_absolute_position_embeddingsposition_embeddingsruhidden_dropout_probrL)r)rarr,s r-r"z!InternVLVisionEmbeddings.__init__s$ ekk!Q8J8J&KL   ll5;;q!V=O=O+PQDO"DO =f E ++&++[__-E-EF   ##V%6%67  ++77  2 2')||EKK;QR?TZTfTf4g'hD $'+D $zz&"<"<= r.rrrc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j dz} ||j dz} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr0r?rrbicubicF)r}mode align_cornersrO)r>rr%jit is_tracingrrr~permuter#rU interpolatercat) r)rrrr num_positionsclass_pos_embedpatch_pos_embedrP new_height new_widthsqrt_num_positionss r-interpolate_pos_encodingz1InternVLVisionEmbeddings.interpolate_pos_encodingsj!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"tq11 T__Q// &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr.rbool_masked_posc|j\}}}}|j|}|j\}}}|K|jj ||d} |j dj | } |d| z z| | zz}|jj |dd} tj| |fd}|j||j|||z}|j|}|S)Nr0rrO) r>rr}rexpand unsqueezetype_asrr%rrrrL) r)rrrrrrrr mask_tokensw cls_tokenss r-r;z InternVLVisionEmbeddings.forwards +001fe**<8 !+!2 GQ  &//00WbIK))"-55kBA#q1u- a?J^^**:r2> YY J7Q?  # # /#d&C&CJPVX]&^^J\\*- r.r) rArBrCrrr"r%rintr BoolTensorr;rDrEs@r-rrs >3>>,&D5<<&D&DUX&D]b]i]i&DV48ll))D0  r.rcVeZdZfdZdej dej fdZxZS)InternVLVisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r!r"rar hidden_act activation_fnr#ror*intermediate_sizefc1fc2r)rar,s r-r"zInternVLVisionMLP.__init__8sd  #F$5$5699V//1I1IJ99V55v7I7IJr.r8rcl|j|}|j|}|j|}|Sr)rrr)r)r8s r-r;zInternVLVisionMLP.forward?s4/ **=9 / r.)rArBrCr"r%rr;rDrEs@r-rr7s$KU\\ellr.r) layer_normrms_normceZdZdZdeddffd ZdejdeejeejejfzfdZ xZ S)InternVLVisionLayerz?This corresponds to the Block class in the timm implementation.rarNct||j|_d|_t ||_t ||_t|j|j|j|_ t|j|j|j|_ |j}tj |t#j$|jzd|_tj |t#j$|jzd|_tj*|j,|_y)Nrr+T) requires_grad)r!r"chunk_size_feed_forward seq_len_dimr` attentionrmlpNORM2FN norm_typer*layer_norm_epslayernorm_beforelayernorm_afterlayer_scale_init_valuer#r$r%r&lambda_1lambda_2rurrL)r)ra init_valuesr,s r-r"zInternVLVisionLayer.__init__Ls '-'E'E$08$V, '(8(8 9&:L:LRXRgRg h&v'7'789K9KQWQfQfg33  [5::f>P>P3Q%Qaef  [5::f>P>P3Q%Qaef zz&"<"<= r.r8c|j|j|\}}|j|z}||z}|j|}|j |}|j |}|j |j |z}||z}|Sr)rrrrrrLr)r)r8attention_outputr layer_outputs r-r;zInternVLVisionLayer.forward[s#nn  ! !- 0 ! ==+;;)=8 ++M: xx - ||L1 == $==<7L$m3 r.) rArBrCrrr"r%rr=r;rDrEs@r-rrIsUI >3 > >|| u|| uU\\5<<%?@ @r.rcReZdZdeddffd Zdej deezfdZ xZ S)InternVLVisionEncoderrarNct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r!r"rar# ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r)rair,s r-r"zInternVLVisionEncoder.__init__xsO  ]]vOgOgIh#iA$7$?#ij &+#$jsA#r8cL|jD] }||} t|S)N)last_hidden_state)rr )r)r8 layer_modules r-r;zInternVLVisionEncoder.forward~s3!JJ 8L(7M 8+  r.) rArBrCrr"r%rr=r r;rDrEs@r-rrws7,3,,  ||      r.rceZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ eedZej$fdZxZS) InternVLVisionPreTrainedModelrainternvl_visionr)imagevideoTr)r8 attentionsc$t||t|trwt j |j |jt j |j|j t j |jyyt|trit j|j|jjt j|j|jjyy)zInitialize the weightsN)r! _init_weightsrrinitzeros_rrrr constant_rrarr)r)rFr,s r-rz+InternVLVisionPreTrainedModel._init_weightss f% f6 7 KK(( )  , F--.))5 F6676  3 4 NN6??DKK,N,N O NN6??DKK,N,N O5r.)rArBrCr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr`_can_record_outputsr%no_gradrrDrEs@r-rrsn  )$O)&*#./N"&-- U]]_ P Pr.rc eZdZdeddffd ZdZede d dejd ejdzde e zfd Z xZS) InternVLVisionModelrarNc2t||||_t||_t ||_|jrtjn*tj|j|j|_ |jy)Nr)r!r"rarrrencoderuse_mean_poolingr#rv LayerNormr*r layernorm post_initrs r-r"zInternVLVisionModel.__init__so   26:,V4 $44BKKM",,vGYGY_e_t_t:u  r.c.|jjSr)rrr?s r-get_input_embeddingsz(InternVLVisionModel.get_input_embeddingss///r.Ftie_last_hidden_statesrrc |j||}|j|}|d}|j|}t||j|j S)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rr)rr8r)rrrrr8r)r)rrrXembedding_outputencoder_outputssequence_outputs r-r;zInternVLVisionModel.forwards` ??<?Y,,'78)!,..93-)77&11  r.r)rArBrCrr"rrrr%rrr=rr;rDrEs@r-rrsm 3  0u5UY !LL ;@;K;Kd;R 5 5 6 r.rc<eZdZUeed<dZdZdZdZdZ dZ dZ dZ dZ y)InternVLPreTrainedModelramodel)rtextrTpast_key_valuesN)rArBrCrrrrr _skip_keys_device_placementr r _can_compile_fullgraphr rrr.r-r"r"s= 1&*#"3N!"&r.r"c*eZdZdeffd ZdZxZS)InternVLMultiModalProjectorrac*t|tj|jj t d|jz dzz|_tj|jj t d|jz dzz|jj |_ t|j|_tj|jj |jj |_y)Nrr)r!r"r#r vision_configr*rdownsample_ratiorro text_configlinear_1rprojector_hidden_actactlinear_2rs r-r"z$InternVLMultiModalProjector.__init__s ,,v';';'G'G#aRXRiRiNiJjnoJo'op  , ,s1v7N7N3N/OST/T TV\VhVhVtVt  &556 &"4"4"@"@&BTBTB`B`a r.c|j|}|j|}|j|}|j|}|Sr)rr.r0r1)r)image_featuresr8s r-r;z#InternVLMultiModalProjector.forwards@7  m4 /  m4 r.)rArBrCrr"r;rDrEs@r-r)r)sb~br.r)zM Base class for InternVL outputs, with hidden states and attentions. c:eZdZUdZdZej dzed<y)InternVLModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)rArBrCrr6r% FloatTensorrrr.r-r5r5s 59**T18r.r5zx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. cheZdZddiZdeffd ZdZdZede d  dd e jd e e e zd zded zdeedeezf dZde j(de jde jfdZede dde j(d zd e jd zde j,d zde j(d zded zde jd zd e e e zd zded zde j(d zdeedeezfdZd de j,defdZxZS)! InternVLModel^language_model.modellanguage_modelract||tj|j|_t ||_tj|j|_ |jyr) r!r"r from_configr+ vision_towerr)multi_modal_projectorr-r;rrs r-r"zInternVLModel.__init__sY  %11&2F2FG%@%H"'33F4F4FG r.c6|jjSr)r;rr?s r-rz"InternVLModel.get_input_embeddingss""7799r.c:|jj|yr)r;set_input_embeddingsr)rIs r-rBz"InternVLModel.set_input_embeddings s 007r.FrzWObtains image last hidden states from the vision tower and apply multimodal projection.rNrvision_feature_layervision_feature_select_strategyrXrc &|j|j}|jj}|dk7rd|d<|jd |dd|}|dk(r |j }n|j |}|dk(r|ddddddf}|jd}t|d z} |jd } |j| | | d}|j|| }|j| d|jd}|j|}||_ |S) a! pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. )r2r0Toutput_hidden_states)r return_dictdefaultNrrr) scale_factorr) r3r2rar,r>rr8r>rr~ pixel_shuffler? pooler_output) r)rrDrErXr,vision_outputsvision_featureschannels feature_sizers r-get_image_featuresz InternVLModel.get_image_features#s:"$TZZ8 ;;77 2 %-1F) ****a RVaZ`a 2 %,>>O,::;OPO )Y 6-aQh7O#((+8S=) $**1- *11*lLZ\],,_K[,\*11*b/BWBWXZB[\44_E'6$r. input_ids inputs_embedsr3cN|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd|jdz}|jdj|j|j }t||j|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )r2devicer0rrz6Image features and image tokens do not match, tokens: z , features: )rr%tensorraimage_token_idlongrUallsumr>r expand_asr3rnumel)r)rRrSr3special_image_maskn_image_tokensn_image_featuress r-get_placeholder_maskz"InternVLModel.get_placeholder_maskUs  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r.rJ position_idsr%cache_positionc |du|duz r td||j|}|k|j|||dj} | j |j |j } |j||| } |j| | }|jd||||| d| } t| j| j| j| j| SdS)Nz:You must specify exactly one of input_ids or inputs_embedsT)rrDrErH)rSr3)rJrar%rSrb)rr%r8rr6r)rirrQrLr3rUr2r`masked_scatterr;r5rr%r8r)r)rRrrJrar%rSrDrErbrXr3r]outputss r-r;zInternVLModel.forwardms<  -t"; <YZ Z  7D557 BM  #!44)%9/M 5 m  ,..}/C/C]EXEXYN!%!:!:~";" *889K^\M%$%% )%+')    +%77#33!//))2>2J   QU   r.rNrJc |j\}}}}||zdk7s||zdk7r td|j||t||zt||z }|j ddddj }|j|t||zt||zt||dzz }|j ddddj }|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)r}rirrrrW)r)rNrJrrrrOs r-rKzInternVLModel.pixel_shuffles/>.B.B.D+ E68 L A %)=)Bjk k*.. s6L#893x,?V;W *11!Q1=HHJ*.. F\12C 8L4MsS[_kmn_nSoOp  *11!Q1=HHJr.NN) NNNNNNNNN)r)rArBrC_checkpoint_conversion_mappingrr"rrBrrr%r7rliststrrrr=rrQ LongTensorr`rrr5r;floatrKrDrEs@r-r9r9 s !"2&"~:8u5n 8<59 ,'',"DIo4,),d , +, , + + ,6,\"))":?:K:K"]b]n]n"0u5.215.204(,267;5926/ ##d*/ ''$./  t+ / &&- /  / ((4// "DIo4/ ),d / ((4// +,/  , ,/ 6/ b!U\\!!r.r9zT Base class for InternVL causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) InternVLCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsr%r8rr6)rArBrCrror%r7rrpr%rr8r=rr6rr.r-rnrns &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r.rnzV The INTERNVL model which consists of a vision backbone and a language model. c teZdZdddddZddiZdeffd Zd Zd Zd e jfd Z e d!de jdeeezdzdedzdeed eezf dZede d"de j0dzde jdzde j2dzde j0dzdedzde jdzdeeezdzdedzde j0dzde j0dzdee j2zde j2dzdeed eezfdZ d#fd ZxZS)$ InternVLForConditionalGenerationzmodel.language_modelzmodel.vision_towerzmodel.multi_modal_projectorlm_head)r:z ^vision_towerz^multi_modal_projectorz^language_model.lm_headzlm_head.weightz(model.language_model.embed_tokens.weightract||t||_t j |j j|j jd|_ |jy)NFrc) r!r"r9r#r#ror-r* vocab_sizersrrs r-r"z)InternVLForConditionalGeneration.__init__sS  "6* yy!3!3!?!?ASASA^A^ejk  r.c6|jjSr)r#rr?s r-rz5InternVLForConditionalGeneration.get_input_embeddingsszz..00r.c:|jj|yr)r#rBrCs r-rBz5InternVLForConditionalGeneration.set_input_embeddingss ''.r.rc|jSr)rsr?s r-get_output_embeddingsz6InternVLForConditionalGeneration.get_output_embeddingss ||r.NrrDrErXc B|jjd|||d|S)N)rrDrEr)r#rQ)r)rrDrErXs r-rQz3InternVLForConditionalGeneration.get_image_featuress5-tzz,, %!5+I    r.FrrRrJrar%rSlabelsrblogits_to_keep image_sizesc  |jd||||||||| | d | }|d}t| tr t| dn| }|j |dd|ddf}d}| 4|j d|| |j jjd| }t|||j|j|j|jS)ac Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```) rRrrJrar%rSrDrErbr}rN)rpr{ru)rorpr%r8rr6r)r#rrslicers loss_functionrar-rurnr%r8rr6)r)rRrrJrar%rSrDrEr{rbr|r}rXrer8 slice_indicesrpros r-r;z(InternVLForConditionalGeneration.forwardsj$**  %)%+'!5+I)#      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD.#33!//)) ' ; ;   r.c ht ||f||||||d| } |s| jdds|| d<| S)N)r%rSrJrbr|is_first_iteration use_cacheTr)r!prepare_inputs_for_generationget) r)rRr%rSrrJrbr|rrX model_inputsr,s r-rz>InternVLForConditionalGeneration.prepare_inputs_for_generationes\w<   +')))1     VZZ T%B ,8L (r.rg) NNNNNNNNNNrN)NNNNNNF)rArBrCrh_tied_weights_keysrr"rrBr#Moduleryrr%r7rrirjrrr=rrQrrkrrrnr;rrDrEs@r-rrrrs=#9.#@$- &" +,VW~ 1/ryy8<59  ''  "DIo4  ),d  +,  + +    u5.215.204(,267;59*.26-.+/S ##d*S ''$.S  t+ S &&- S  S ((4/S "DIo4S ),d S   4'S ((4/S ell*S \\D(S +,S  / /S 6S p   r.rr)rrr"r9rr)r|)Bcollections.abcrr dataclassesrr%torch.nnr#rr activationsr cache_utilsr generationr integrationsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrautorconfiguration_internvlrrrrrrlr^r`rrrrrrrrrrr"r)r5r9rnrr__all__rr.r-rs,$! &! )79ddF&gg/HY'JBIIJ(J6% II% <<% % << % LL4' %  %%6F$biiF$R  +E   BII J[ryy[|  3H I+4+\ BII &POPP@% 7% % P 'o ' '"))$  9"9 9  9 q+q  qh  9[9  90 ^'>^  ^B r.