iddlZddlmZddlmZddlmZddlZddlm Z ddl m cm Z ddl mZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZmZdd lmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7m8Z8ddl9m:Z:m;Z;e4jxe=Z>Gdde j~Z@Gdde j~ZAGddejj~ZBGdde j~ZCGdd e jZEed!Gd"d!e j~ZFGd#d$e j~ZGd%ejd&ejd'ejd(eIejejffd)ZJd*ejd+eKd(ejfd,ZL d_d-e j~d.ejd/ejd0ejd1ejdzd2eMd3eMfd4ZN d_d-e j~d.ejd/ejd0ejd1ejdzd2eMd3eMfd5ZOGd6d7e j~ZPGd8d9e!ZQe2Gd:d;e-ZRe2Gd<d=eRZSGd>d?eReZTee2d@AGdBdCe'ZUGdDdEejj~ZVGdFdGe j~ZWdHZXGdIdJe j~ZYdKejd.ejfdLZZd.ejd/ejdKejd(eIejejffdMZ[GdNdOe j~Z\GdPdQe j~Z]GdRdSe!Z^GdTdUe j~Z_GdVdWe j~Z`GdXdYe j~ZaGdZd[eRZbGd\d]eReZcgd^Zdy)`N)Callable) dataclass)Optional)Llama4VisionConfig)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_maskcreate_chunked_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)check_model_inputsmaybe_autocast) Llama4ConfigLlama4TextConfigc\eZdZdeffd Zdej dej fdZxZS)Llama4TextExpertsconfigct||j|_|j|_|j |_|j|_tjtj|j|j d|j z|_ tjtj|j|j |j f|_ t|j|_yN)super__init__num_local_experts num_expertsintermediate_size hidden_size expert_dimnn Parametertorchzeros gate_up_projempty down_projr hidden_actact_fnselfr( __class__s t/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/llama4/modeling_llama4.pyr-zLlama4TextExperts.__init__8s !33!'!9!9!--00LLT5E5EtGWGWYZ]a]l]lYl)mnekk43C3CT__VZVfVf2g&hiV../ hidden_statesreturncv|j|jjdd|j}t j ||j}|j dd\}}t j ||j|z|j}|jd|j}|S)a2 This should really not be run on a single machine, as we are reaching compute bound: - the inputs are expected to be "sorted" per expert already. - the weights are viewed with another dim, to match num_expert, 1, shape * num_tokens, shape Args: hidden_states (torch.Tensor): (batch_size * token_num, hidden_size) selected_experts (torch.Tensor): (batch_size * token_num, top_k) routing_weights (torch.Tensor): (batch_size * token_num, top_k) Returns: torch.Tensor rr+dim) viewr7shaper1r5bmmchunkr;r9)r=rAgate_upgateup next_statess r?forwardzLlama4TextExperts.forwardBs&**4+<+<+B+B1+Er4K[K[\ ))M4+<+<====+biidkk$&7!7$..I !&&r4+;+;< r@) __name__ __module__ __qualname__r%r-r5TensorrO __classcell__r>s@r?r'r'7s+0/0U\\ellr@r'c&eZdZdfd ZdZxZS) Llama4TextMLPcft|| |j}||_t j |j |d|_t j |j |d|_t j ||j d|_ t|j|_ yNFbias) r,r-r0r(r3Linearr1 gate_projup_projr9r r: activation_fn)r=r(r0r>s r?r-zLlama4TextMLP.__init__Ys   $ & 8 8  6#5#57HuUyy!3!35FUS #4f6H6HuU#F$5$56r@c|j|j||j|z}|j|SN)r_r]r^r9)r=xr9s r?rOzLlama4TextMLP.forwardes7&&t~~a'89DLLOK ~~i((r@rarPrQrRr-rOrTrUs@r?rWrWXs  7)r@rWc8eZdZddeffd ZdZdZdZxZS)Llama4TextL2Normepsc0t|||_yra)r,r-rf)r=rfr>s r?r-zLlama4TextL2Norm.__init__ks r@c|tj|jdjdd|jzzSNr+rDT)keepdimr5rsqrtpowmeanrfr=rbs r?_normzLlama4TextL2Norm._normo45;;quuQx}}R}>IJJJr@c^|j|jj|Sra)rpfloattype_asros r?rOzLlama4TextL2Norm.forwardrs"zz!'')$,,Q//r@c d|jS)Nzeps=rfr=s r? extra_reprzLlama4TextL2Norm.extra_reprusdhhZ  r@)gư>) rPrQrRrsr-rprOrxrTrUs@r?rerejsEK0!r@rec2eZdZdfd ZdZdZdZxZS)Llama4TextRMSNormct|||_tjt j ||_y)z< Llama4RMSNorm is equivalent to T5LayerNorm N)r,r-rfr3r4r5onesweight)r=r1rfr>s r?r-zLlama4TextRMSNorm.__init__zs0 ll5::k#:; r@c|tj|jdjdd|jzzSrirkros r?rpzLlama4TextRMSNorm._normrqr@c||j|jj|}||jzSra)rprsrtr})r=rboutputs r?rOzLlama4TextRMSNorm.forwards0AGGI&..q1 ##r@c^t|jjd|jS)Nz, eps=)tupler}rHrfrws r?rxzLlama4TextRMSNorm.extra_reprs' ))*+6$((<)rPrQrRr-rprOrxrTrUs@r?rzrzys<K$=r@rzc(eZdZfdZfdZxZS) Llama4Routerct||j|jd|j|_|j |_yrY)r,r-r1r.r/num_experts_per_toktop_kr<s r?r-zLlama4Router.__init__s> ++V-E-EER!33// r@ctt||}tj||jd\}}tj |t djd||}tjjj|j j|j}||fS)Nr#rEz-inf) r,rOr5topkr full_likersscatter_r3 functionalsigmoidtodtype)r=rA router_logitsrouter_top_valuerouter_indices router_scoresr>s r?rOzLlama4Router.forwards 6 +0::mTZZUV+W(. uV}ENNqR`brs ++33M4G4G4IJMMmNaNab m++r@rcrUs@r?rrs0 ,,r@r Llama4TextMoec$eZdZfdZdZxZS)rct||j|_|j|_|j |_t||_ t||_ t||_ yra)r,r-rrr1 hidden_dimr.r/r'expertsrrouterrW shared_expertr<s r?r-zLlama4TextMoe.__init__s[ //  ,,!33(0 "6* *62r@c|jd|j}|j|\}}|j|jdd}||j ddjddz}|j |}|j|}|j|j|jdd|jdjd||fS)NrDr#rrE) reshaperrrepeatrH transposerradd_sum)r=rArr routed_in routed_outouts r?rOzLlama4TextMoe.forwards%--b$//B '+{{='A$ }!(()<)s r?r-z"Llama4TextRotaryEmbedding.__init__s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuUr@rz torch.deviceseq_lenrBz torch.Tensorc|jd}t|ddxs|j|jz}d}d|t j d|dtj j|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimN?rr+r)rr) rgetattrr1num_attention_headsr5arangeint64rrs)r(rrbaserFattention_factorrs r?rz9Llama4TextRotaryEmbedding.compute_default_rope_parameterss&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k )))r@cv|jddddfjj|jddd}|dddddfj}t |j j tr/|j j dk7r|j j nd}t|d5|j|j |zjdd}tjtj||}||jz}ddd|S#1swYSxYw) NrrDr#mpscpuF) device_typeenabledr+)rrsexpandrH isinstancertypestrr"rrr5polar ones_liker)r=rb position_idsinv_freq_expandedposition_ids_expandedrfreqs freqs_ciss r?rOz!Llama4TextRotaryEmbedding.forwards!MM$4-8>>@GG HZHZ[\H]_acde ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk  U C ;&))!((36KKVVWXZ[\E EOOE$:EBI!D$:$::I ;   ; s =A'D..D8ra)NNN)rPrQrRr5rS__annotations__r%r- staticmethodrintrrsrno_gradrrOrTrUs@r?rrsllV/V *.+/"* 4'*(*t* ~u$ % **<U]]_  r@rxqxkrrBc &tj|jjg|jdddd}tj|jjg|jdddd}tj ||dddddddfzj d}tj ||dddddddfzj d}|j||j|fS)NrDr+r)r5view_as_complexrsrrH view_as_realflattenrt)rrrxq_xk_xq_outxk_outs r?apply_rotary_embrs    2 2 2 IBHHSbM I2 Iq I JC    2 2 2 IBHHSbM I2 Iq I JC   i1dA &> > ? G G JF   i1dA &> > ? G G JF >>" v~~b1 11r@rAn_repc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r#N)rHrr)rArbatchnum_key_value_headsslenrs r? repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr@modulequerykeyvalueattention_maskscalingdropoutc t||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| d} t jj| ||j} tj| | } | j ddj} | | fS)Nr+rrDrEptrainingr#) rnum_key_value_groupsr5matmulrrHr3rsoftmaxrr contiguous rrrrrrrkwargs key_states value_states attn_weights causal_mask attn_outputs r?eager_attention_forwardr s3 ; ;L==((6??([L,,|\:K''1-88:K  $$r@c t||j}t||j} tj||j dd|j dzz} |#|ddddddd|j df} | | z} tjj| d} tjj| ||j} tj| | } | j ddj} | | fS) Nr+rrrDrErr#) rrr5rrrrHr3rrrrrrs r?vision_eager_attention_forwardr(s3 ; ;L==((6??([L,,|\:K''1-88:K  $$r@c2eZdZdZdeffd Z d dejdeejejfdejdzde dzd ejdzd e e d eejejdzeejdzffd Z xZS)Llama4TextAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr(ct|||_||_t |d|j |j z|_|j |_|j |jz|_ |j|_|jdz|_ |j|_ |j|_ |j|_ |j|_d|_|j ||_t%j&|j |j |jz|j(|_t%j&|j |j|jz|j(|_t%j&|j |j|jz|j(|_t%j&|j |jz|j |j(|_|jj2r(|j"rt5|j6|_yyy)NrrTrZ)r,r-r( layer_idxrr1rrrrr attn_scale floor_scaleattn_temperature_tuningattention_dropout is_causalno_rope_layersuse_roper3r\attention_biasq_projk_projv_projo_proj use_qk_normre rms_norm_epsqk_normr=r(rr>s r?r-zLlama4TextAttention.__init__Es  " F4F4F&JdJd4de #)#=#= $*$>$>&B\B\$\!#)#=#= }}d*  ++!--'-'E'E$!'!9!9--i8 ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf   ;; " "t}}+F,?,?@DL(5 "r@NrAposition_embeddingsrpast_key_valuescache_positionrrBc H|jdd}g|d|j}|j|j|} |j |jg|d|j} |j |j|j dd} |jr)t| | |j| j\} } t|dr"|j| } |j| } |jr|jstjtj |j#dz|j$z |j&zdz} | jd|dddfj)g|dd} | | zj| j*} | j dd} | j dd} |%d|i} |j-| | |j.| \} } t1j2|j4j6t8}||| | | |f|j:sdn |j<|j>d|\}}|j@g|djC}|jE|}||fS) NrDr#r+rrr)rr)#rHrrrGrrrr rrrhasattrrrr5log1pfloorrsrrrrupdaterr get_interfacer(_attn_implementationrrr rrrr)r=rArrrrr input_shape hidden_shape query_statesrr attn_scales cache_kwargsattention_interfacerrs r?rOzLlama4TextAttention.forwardcs$))#2. 88b8$--8 {{=166|D 4T[[/44UkU2Ut}}U {{=166|DNNqRST =='7j*=*@*@ATAT*U( $L* 4 #<< 5Lj1J  ' '  EKK)=)=)?#)EIYIY(YZ[^b^m^mmpss &**A{21+EFMMNbP[Nb]^Nb`aNbcK(;6::<;M;MNL#--a3 ))!Q/  &,n=L'6'='=j,X\XfXfht'u $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r@NN)rPrQrR__doc__r%r-r5rSrr LongTensorrrrOrTrUs@r?rrBsGA/AF)-26 9)||9)#5<<#=>9) t+ 9)  9) ((4/ 9)-.9) u||U\\D0% 2E2LL M9)r@rcZeZdZfdZ d dej dej dzdej dzdedzdedzdej dzd e ej ej fdzd e e d e eje ejejfdzffd Z xZS)Llama4TextDecoderLayerct||j|_||_|j||_t |||_||jv|_ |jrt||_ nt||j|_ t|j|j|_t|j|j|_y)N)r0rv)r,r-r1r layer_typesattention_typer self_attn moe_layers is_moe_layerr feed_forwardrWintermediate_size_mlprzrinput_layernormpost_attention_layernormrs r?r-zLlama4TextDecoderLayer.__init__s !--"$00;,VY?%):)::    -f 5D  -fHdHd eD 01C1CI\I\](9&:L:LRXReRe(f%r@NrArrr use_cacherrrrBc |} |j|}|jd||||||d|\} } | | z}|} |j|}|j|}|jr|\}} | |j | j z}|S)N)rArrrr6r)r4r/r5r2r1rGrH) r=rArrrr6rrrresidualattention_states_s r?rOzLlama4TextDecoderLayer.forwards!,,]; -dnn ' 3)+)   !!#33 !55mD ))-8   , M1 =#5#5hnn#EE r@)NNNFNN)rPrQrRr-r5rSr)r boolrrr FloatTensorrOrTrUs@r?r+r+s g$/304(,!&26HL"||" t+"&&- "  " $; "((4/"#5<<#=>E"-." u  %(9(95;L;L(L"MPT"TT U"r@r+cteZdZUeed<dZdZdgZdZdZ dZ dZ dZ e jfdZxZS)Llama4PreTrainedModelr()imagetextTrFc4t||t|jdr|jjn|jj j}t |trEtj|jd|tj|jd|yt |trWtj|j|jtj|j|jyy)Ninitializer_ranger)rnstd)rD)r, _init_weightsrr(rC text_configrr'initnormal_r7r9Llama4VisionModelclass_embeddingscalepositional_embedding_vlm)r=rrDr>s r?rEz#Llama4PreTrainedModel._init_weightss f%t{{$78 KK ) )((:: f/ 0 LL,,3C @ LL)) =  1 2 LL//V\\ B LL88fll K3r@)rPrQrRr$rinput_modalitiessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendr5rrErTrUs@r?r?r?sW (&*##4"5 N!"&U]]_ L Lr@r?c>eZdZUdgZdZdZeed<ee e dZ deffd Z e ee ddej"dzd ej$dzd ej"dzd edzd ej(dzd edzdej"dzdeedeezfdZxZS)Llama4TextModelr+model)rAr() attentionsrArc  t|||j|_|j|_t j |j|j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_t#||_d|_|j)ycc}w)Nrvr(F)r,r- pad_token_id padding_idx vocab_sizer3 Embeddingr1 embed_tokens ModuleListrangenum_hidden_layersr+layersrzrnormr rotary_embgradient_checkpointing post_initrs r?r-zLlama4TextModel.__init__s  !.. ++LL):):FQ>QR 36B&+#  isDN input_idsrrr inputs_embedsr6rrrBc |du|duz r td|>|j|j|jjj}|r|t |j }|F||jnd} tj| | |jdz|j}||jd}t|x} ts*|j |||||d} td i| td i| d} |} |j!| |} |j"d|j j$D]}|| f| |j&||||| d|} !|j)| } t+| |r| Sd S) N:You must specify exactly one of input_ids or inputs_embedsrZrr#)r)r( input_embedsrrrr)full_attentionchunked_attention)rrrr6rr)last_hidden_staterr8) ValueErrorr_rr}rr r(get_seq_lengthr5rrH unsqueezerdictrrrercrbr.rdr)r=rhrrrrir6rrpast_seen_tokenscausal_mask_mapping mask_kwargsrAfreq_cis decoder_layers r?rOzLlama4TextModel.forward s  -t"; <YZ Z   --ill4;L;L;S;S;Z;Z.[\M 0*$++>O  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L?-F++ -"0"0#2 , K#5"C{"C%?%N+%N#  & ??=,?![[)H4;;+H+HI M) 2=3O3OP) /#-$,  M  -0 &+/8O  >B  r@)NNNNNNN)rPrQrR_no_split_modulesbase_model_prefixrMr%rrr+r_can_record_outputsr-rr!rr5r)rSr r=r<rrrrrOrTrUs@r?rVrVs#12  )/& / .2.204(,26!%26C ##d*C  t+C &&- C  C ((4/ C $;C ((4/C +,C  ( (C C r@rVcteZdZUdgZdZddiZddiZeed<deffd Z e e dd e jd zd e jd zd e jd zd ed zde j d zde jd zded zde jd zdee jzdeedeezfdZxZS)Llama4ForCausalLMr+language_modelzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr(ct||t||_|j|_t j |j|jd|_|jyrY) r,r-rVrWr]r3r\r1rrgr<s r?r-zLlama4ForCausalLM.__init__]sU  $V,  ++yy!3!3V5F5FUS  r@Nrhrrrrilabelsr6rlogits_to_keeprrBc l|jd|||||||d| } | d} t| tr t| dn| } |j | dd| ddf}d}|*|j d|||j jd| }t||| j| j| jS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Llama4ForCausalLM >>> model = Llama4ForCausalLM.from_pretrained("meta-llama4/Llama4-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-llama4/Llama4-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rhrrrrir6rrN)logitsrr])lossrrrArXr8) rWrrslicer loss_functionr(r]rrrArX)r=rhrrrrirr6rrroutputsrA slice_indicesrrs r?rOzLlama4ForCausalLM.forwardfsJ$**  )%+')      8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD%#33!//))   r@) NNNNNNNNr)rPrQrRryrz_tied_weights_keys_tp_planr%rr-rrr5r)rSr r=r<rrrrrrOrTrUs@r?r}r}Vs>12(*,GH23H /.2.204(,26*.!%26-.< ##d*<  t+< &&- <  < ((4/ <   4'< $;< ((4/< ell*< +,<  ' '< < r@r}zQ Base class for Llava causal language model (or autoregressive) outputs.  custom_introceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) Llama4CausalLMOutputWithPasta3 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size (batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. NrrrrArXimage_hidden_states)rPrQrRr(rr5r=rrrr rArrXrr8r@r?rrs &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r@rc$eZdZfdZdZxZS)Llama4VisionMLP2c~t||j|_|j|_t j |j|j d|_t j |j|jd|_ t j|_ |j|_ yrY)r,r-r1r0r3r\projector_input_dimfc1projector_output_dimfc2GELUr_projector_dropoutrr<s r?r-zLlama4VisionMLP2.__init__s !--!'!9!999T33V5O5OV[\99V88&:U:U\abWWY// r@c|j|}|j|}tj||j|j}|j|j |S)Nr)rr_Frrrr=rAs r?rOzLlama4VisionMLP2.forwardsT/ **=9  -4<<$--X !!$((="9::r@rcrUs@r?rrs 0;r@rc$eZdZfdZdZxZS)Llama4MultiModalProjectorct|tj|jj |j jd|_yrY) r,r-r3r\ vision_configvision_output_dimrFr1linear_1r<s r?r-z"Llama4MultiModalProjector.__init__s?   2 2    * *  r@c(|j|}|Sra)r)r=image_featuresrAs r?rOz!Llama4MultiModalProjector.forwards n5 r@rcrUs@r?rrs  r@rc J|j\}}}ttj|}|j |||d}|j \}}}}|j ||t||zt||z }|j ddddj}|j |t||zt||zt||dzz }|j ddddj}|j |d|jd} | S)NrDrr+r#r)rHrmathsqrtrGsizepermuter) input_tensor shuffle_ratio batch_size num_patcheschannels patch_sizeheightwidthreshaped_tensor output_tensors r? pixel_shufflers%(4(:(:%J XTYY{+,J$$ZZLL*6*;*;*='Jx"'' FC @UUYZ r@NrArrrrrBc |jdd}g|d|j}|j|j|}|j |j|} |j |j|} t || |\}} |jdd}| jdd} | jdd} tj|jjt} | ||| | df|jsdn |jddd|\} } | jg|dj!} |j#| } | | fS)NrD)rr#r+rF)rrr )rHrrrGrrrrrrr(r rrr rrr)r=rArrrrr!r"r#rrr&rrs r?rOzLlama4VisionAttention.forward*sj$))#2. 88b8$--8 {{=166|D [[/44\B {{=166|D #:<^f#g j#--a3 ))!Q/ #--a3 (?(M(M KK , ,.L) %8      % $}}C$2H2H %  % ! \*k));;;;FFH kk+. L((r@r') rPrQrRrr-r5rSr rrrrOrTrUs@r?rrs [1 [&/3(, ')||'),,') t+ ')  ') -. ') u||U\\D0% 2E2LL M')r@rcVeZdZfdZdej dej fdZxZS)Llama4VisionMLPc&t|||_tj|_tj |j|jd|_ tj |j|jd|_ y)NTrZ) r,r-r(r3rr_r\r1r0rrr<s r?r-zLlama4VisionMLP.__init__Use  WWY99V//1I1IPTU99V55v7I7IPTUr@rArBcl|j|}|j|}|j|}|Sra)rr_rrs r?rOzLlama4VisionMLP.forward\s4/ **=9 / r@rrUs@r?rrTs$VU\\ellr@rc eZdZdeffd Z d dej dej dej dzdedzfdZxZ S) Llama4VisionEncoderLayerr(ct||j|_t||_t ||_tj|j|_ tj|j|_ yra) r,r-r1rr/rrr3 LayerNormr4r5r<s r?r-z!Llama4VisionEncoderLayer.__init__dsb !--.v6"6*!||F,>,>?(* V5G5G(H%r@N hidden_staterroutput_attentionsc|}|j|}|j|||\}}||z}|}|j|}|j|}||z}|f}|r||fz }|S)N)rr)r4r/r5r)r=rrrrr9rrs r?rOz Llama4VisionEncoderLayer.forwardns ++L9 %)^^ )&4& " l  ,.  44\B xx - ,. /    &Gr@r') rPrQrRrr-r5rSr<rOrTrUs@r?rrcsZI1I/3)- ll,, t+   $; r@rceZdZdZdeffd Z d dejdejdejdzdedzd edzd edzd e e zfd Z xZ S)Llama4VisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Llama4VisionEncoderLayer`]. Args: config: Llama4VisionConfig r(ct|||_tjt |j Dcgc] }t|c}|_d|_ ||_ycc}w)NF) r,r-r(r3r`rarbrrcrf)r=r(r;r>s r?r-zLlama4VisionEncoder.__init__sW  mmuU[UmUmOn$o!%=f%E$op &+# %psA*NrArrroutput_hidden_states return_dictrBcz||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|jD]&} |r||fz}| ||||} |r || dfz}| d}(|r||fz}|st d|||fDSt |||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr8)rrrrr#rc3&K|] }|| ywrar8.0vs r? z.Llama4VisionEncoder.forward..seqWXWderorArX)r(rruse_return_dictrcrr) r=rArrrrrencoder_statesall_attentions encoder_layer layer_outputss r?rOzLlama4VisionEncoder.forwards>2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r@NNNN) rPrQrRr(rr-r5rSr<rrrOrTrUs@r?rrs1/3)-,0#'? ||? ,,?  t+ ?  $; ? #Tk ? D[?   ? r@rcVeZdZfdZdej dej fdZxZS)Llama4UnfoldConvolutionc<t||j}t|tr||f}t j j||j|_t j|j|dz|dz|jd|_ y)N) kernel_sizestriderr#FrZ) r,r-rrrr5r3Unfoldunfoldr\ num_channelsr1linear)r=r(rr>s r?r-z Llama4UnfoldConvolution.__init__s '' k3 '& 4Khhoo+fFWFWoX ii   +a. 0;q> A     r@rArBcp|j|}|jddd}|j|}|S)Nrr+r#)rrrrs r?rOzLlama4UnfoldConvolution.forwards8 M2 %--aA6  M2 r@rrUs@r?rrs#  U\\ellr@rc*eZdZdeffd ZdZxZS)Llama4VisionRotaryEmbeddingr(ct ||j|jz}t j |dztj j|dzd}t j||ddgd}d|d<||z}||z}|j|jzdz}d|jd t j d|dd|dzj|z zz }|dzd |ddddfzjdd }|dzd |ddddfzjdd } t j|| gd jjd dddf} | j|jd dddkd} t j t j"t j$| t j&| gd } | |_y) Nr+rr#rrEr)rDrDrr).NrD.)r,r- image_sizerr5rint32rcatr1rrrsrepeat_interleaver masked_fillrstackcossinr) r=r(idximg_idx frequencies_x frequencies_yfreq_dim rope_freqfreqs_xfreqs_yrrwr>s r?r-z$Llama4VisionRotaryEmbedding.__init__s 6#4#44,,sAvU[[9AA#q&!L))Wgbqk2:# 3 %%)C)CCqH  " "< 0 Q!,->A ?EEG(R T "A%y1IdD!m4LL__`agi_j!A%y1IdD!m4LL__`agi_j 7G,"5;;=HHJ3PSRSPS8T!!'//"a";a"?C((eii6F RWHX5Y_a)bc  r@cL|jj|jSra)rrrrs r?rOz#Llama4VisionRotaryEmbedding.forward s}} 4 455r@)rPrQrRrr-rOrTrUs@r?rrs!1!(6r@rceZdZUdZdZdgZeed<deffd ZdZ dde jd e jdzd e dzd e dzd e dzd e ee jdfzf dZxZS)rI vision_model)r@rr(crt|||j|_|j|_|j|_|j |_|j|jzdzdz|_|jdz|_t||_ tj|jtj|jz|_tj|jtj|j |jz|_t!||_tj$|j|_tj$|j|_t+||_t/||_|j3y)Nr+r#r)r,r-rrr1rrrKrpatch_embeddingr3r4r5randnrJrLrrotary_embeddingr layernorm_prelayernorm_postrrWrvision_adapterrgr<s r?r-zLlama4VisionModel.__init__sA   ++ ++!--"// OOt>1DqH''- 6v>!||DJJTEUEU9V,VW(* TZZ%++dN^N^`d`p`pBq5q(r% ;F C \\$*:*:; ll4+;+;<)0 9&A r@c|jS)zg This function is used to fetch the first embedding layer to activate grads on inputs. )rrws r?get_input_embeddingsz&Llama4VisionModel.get_input_embeddings-s###r@N pixel_valuesrrrrrB.c ||n|jj}||n|jj}||n|jj}|j\}}} } d} d} |j |} | j\}}}| j || z| z||} |jj| jdd| jd}tj| |gd} |dz }| j || z| ||} |jj| j| j}| |z} |j| } | j!|d|} |j#|}|j%| d|||}|j&} |j)| } | ddddddf} |j+| } |r |j,nd}|r|d}nd}|st/d | ||fDSt1| || S) a Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, MllamaVisionModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaVisionModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 1, 4, 1025, 7680]) ``` Nr#rrDrErr)rrrrr+c3&K|] }|| ywrar8rs r?rz,Llama4VisionModel.forward..s_qQRQ^_rr)r(rrrrHrrrJrr5rrLrrrrrGrrWrorrrArr)r=rrrrrrbatch_size_times_num_tilesrrrnum_concurrent_media num_chunksrr;rrrJpositional_embeddingrrrArXs r?rOzLlama4VisionModel.forward3sTD2C1N-TXT_T_TqTq$8$D $++JjJj &1%v%F"/0B0BC ,,778< 8P8P8\DKK44bd r@c6|jjSra)r~rrws r?rz3Llama4ForConditionalGeneration.get_input_embeddingss""7799r@c:|jj|yra)r~set_input_embeddings)r=rs r?r(z3Llama4ForConditionalGeneration.set_input_embeddingss 007r@c6|jjSra)r~get_output_embeddingsrws r?r*z4Llama4ForConditionalGeneration.get_output_embeddingss""88::r@c:|jj|yra)r~set_output_embeddings)r=new_embeddingss r?r,z4Llama4ForConditionalGeneration.set_output_embeddingss 11.Ar@c:|jj|yra)r~ set_decoder)r=decoders r?r/z*Llama4ForConditionalGeneration.set_decoders ''0r@c6|jjSra)r~ get_decoderrws r?r2z*Llama4ForConditionalGeneration.get_decoders""..00r@F)tie_last_hidden_stateszOObtains image last hidden states from the vision tower and apply al projection.rrvision_feature_select_strategyrrBc |jDcic] \}}| || }}}|j|fi|Scc}}w)aj pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_select_strategy (`str`): The feature selection strategy used to select the vision feature from the vision backbone. Can be one of `"default"` or `"full"` )itemsr)r=rr4rkrs r?get_image_featuresz1Llama4ForConditionalGeneration.get_image_featuressG$*<<>C41aQ]!Q$CC t  888Ds ::rhrirc*|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jdj|j|j }t||j|jk(d|d|jd|S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. rrDz6Image features and image tokens do not match, tokens: z , features: r)rr5tensorr(image_token_idlongrallrrr expand_asrr numelrH)r=rhrirspecial_image_maskn_image_tokenss r?get_placeholder_maskz3Llama4ForConditionalGeneration.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aoauauvwax`y z "!r@Nrrrrr6rrrrrc | | n|jj} | | n|jj} | | n|jj} |du|duz r t d| | t d||j |}||j ||dj}|jd|jd}|j|j|j|j}|j|||}|j||}|j d ||||| | | | | |d |}|d }d}|<||dd|j"d d z dfj|j}|d ddddf|j|jd k7j%}|d d df|j|jd k7j%}n1|d ddddfj%}|d d dfj%}t'j(}||jd|jd|jdj|j}| s|f|d dz}||f|zS|St+|||j,|j.|j0| Sd S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, LlavaForConditionalGeneration >>> model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> prompt = "USER: \nWhat's the content of the image? ASSISTANT:" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "USER: \nWhat's the content of the image? ASSISTANT: The image features a busy city street with a stop sign prominently displayed" ```NrkzdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either oneT)rr4rrD)rir) rrrrir6rrrrrrr#.)rrrrArXrr8)r(rrrrprr8 pooler_outputrGrr%rrrrBmasked_scatterr~rHrr3CrossEntropyLossrrrArX)r=rhrrrrrir4rr6rrrrrrr vision_flatprojected_vision_flatr@rrrshift_attention_mask shift_logits shift_labelsloss_fctrs r?rOz&Llama4ForConditionalGeneration.forwardsSd2C1N-TXT_T_TqTq$8$D $++JjJj &1%$>{$K$N$N$$m&9&9% !"&!:!:G\";" *889KMbcM%$%%  )%+'/!5#))      )(6a6<<?Q;N9O9Q6Q'R'U'UV\VcVc'd$%c3B3k23G3J3J6==3Y]^3^_jjl %c12g/C/F/Fv}}/UYZ/Z[ffh %c3B3k2==? %c12g99; **,H!!"l&7&7&;O>OPR>S>V>VWcWjWj>kDY,F'+'7D7V# CV C+#33!//))2>2J   QU   r@c |jj|f||||||d| } |s| jdds|| d<| S)N)rrirrris_first_iterationr6Tr)r~prepare_inputs_for_generationget) r=rhrrirrrrrNr model_inputss r?rOzrks $! N&!.)7KB9G&jj?@   H % B)BII)$ !uxx != =( ,299 ,_-"BII".",? ?D 2  2  2|| 2 5<< %& 2 UU\\ U# U%,, U(% II% <<% % << % LL4' %  %%D% II% <<% % << % LL4' %  %%4Z)"))Z)z272jLOLL8a +a a HN -N b  9;9  90;uxx;"  ( )")) )!ELL!! : << :  :ll : 5<< %& :7)BII7)t bii )9)XO "))O dbii(6"))62G -G Tn%:Onb r@