imDddlZddlmZmZddlmZddlmZddlZddl m Z ddl m cm Z ddlmZddlmZddlmZmZdd lmZdd lmZdd lmZmZdd lmZdd lm Z m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5ddl6m7Z7m8Z8m9Z9m:Z:ee.Gdde!Z;ee.dGdde ZZOd?ejd@eQdAejfdBZR dhdCe j|dDejdEejdFejdGejdzdHeSdIeSdzdJeSdzdAeTejejffdKZUdidLejdMejdNejdOeQfdPZVeeVGdQdRe j|ZWGdSdTeZXe.GdUdVe(ZYGdWdXe j|ZZe.dYGdZd[eYZ[e.d\Gd]d^eYeZ\Gd_d`e j|Z]e.daGdbdceYZ^e.ddGdedfeYeZ_gdgZ`y)jN)CallableSequence) dataclass)Optional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernelized_func)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)check_model_inputsmaybe_autocast) AutoModel)Gemma3nAudioConfig Gemma3nConfigGemma3nTextConfigGemma3nVisionConfigc:eZdZUdZdZej dzed<y)Gemma3nAudioEncoderModelOutputzz audio_mel_mask (`torch.FloatTensor`, *optional*): A torch.BoolTensor of shape `(batch_size, num_frames)` Naudio_mel_mask)__name__ __module__ __qualname____doc__r)torch BoolTensor__annotations__v/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/gemma3n/modeling_gemma3n.pyr(r(5s /3NE$$t+2r2r(zL Base class for Gemma3n outputs, with hidden states and attentions.  custom_introcbeZdZUdZdZej dzed<dZej dzed<y)Gemma3nModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. audio_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. audio_hidden_states of the model produced by the audio encoder and after projecting the last hidden state. Nimage_hidden_statesaudio_hidden_states) r*r+r,r-r8r. FloatTensorr0r9r1r2r3r7r7@s5 59**T1848**T18r2r7zS Base class for Gemma3n causal language model (or autoregressive) outputs. c"eZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<dZej dzed <y) Gemma3nCausalLMOutputWithPastaF loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. audio_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. audio_hidden_states of the model produced by the audio encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr8r9)r*r+r,r-r=r.r:r0r>r?r r@tuplerAr8r9r1r2r3r<r<Zs$&*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T1848**T18r2r<creZdZd dededeffd ZdZdejdejfdZ d Z xZ S) Gemma3nRMSNormdimeps with_scalect|||_||_|jr.t j t j||_y|jdt jddy)Nweight?F persistent) super__init__rFrGnn Parameterr.onesrIregister_buffertensor)selfrErFrG __class__s r3rNzGemma3nRMSNorm.__init__~sY $ ??,,uzz#7DK  5<<+< Or2c|tj|jdjdd|jzz S)Nr T)keepdim)r.sqrtpowmeanrF)rTxs r3_normzGemma3nRMSNorm._norms45::aeeAhmmBm=HIIIr2r\returnc|j|j|jjz}|j|SN)r]floatrItype_as)rTr\outputs r3forwardzGemma3nRMSNorm.forwards9AGGI&):):)<<~~a  r2c^t|jjd|jS)Nz, eps=)rBrIshaperFrTs r3 extra_reprzGemma3nRMSNorm.extra_reprs' ))*+6$((<T) r*r+r,intraboolrNr]r.Tensorrdrh __classcell__rUs@r3rDrD}sGPCPePPJ!!%,,! =r2rDceZdZdeffd Zdej dejdej fdZdej de d e d e d e d e d e dej fdZ dej dej dej fdZ xZ S)%Gemma3nAudioRelativePositionEmbeddingconfigcRt|||_|jj|_|jj |_|j |jz|_td|jjdz |_ |jj|_ tj|j |j|jzd|_d}d}|j dz}t!j"t%|t%|z t|dz dz }|t'j(t'j*|| zz}|j-d|j%j/dj/dd y) Nrr"FbiasrJ@r inv_timescalesrK)rMrNrpconf_num_attention_heads num_heads hidden_sizechannelshead_dimmaxconf_attention_context_left max_backwardconf_attention_context_right max_forwardrOLinearpos_projmathlograr.exparangerR unsqueeze)rTrp min_timescale max_timescalenum_timescaleslog_timescale_incrementrurUs r3rNz.Gemma3nAudioRelativePositionEmbedding.__init__sL  == //  7 4;;#J#JQ#NO;;CC $--$--1OV[\   !+"&((5+?% BV+V"WZ]^lop^prsZt"t&5<<3OSjRj3j)kk    " , ,Q / 9 9! <  r2positiondtyper^cP|jjd}||jj|jt j z}t jt j|t j|gd}|j|S)NrWdevicerrE) rarrutorr.float32catsincostype)rTrr scaled_time timing_signals r3_get_timing_signal_1d_posz?Gemma3nAudioRelativePositionEmbedding._get_timing_signal_1d_poss}>>#--b1!4!4!7!7xV[VcVc!7!dd  599[#9599[;Q"RXZ[ !!%((r2term_bd_before_shift batch_sizerwnum_query_blocksquery_block_sizekey_context_sizemax_span_plus_1c|dz|z }d|f} tjj|| } | j|||||dzzf} | ddddddd||zf} | j|||||f} | S)aZPerforms the relative shift. Args: term_bd_before_shift: Tensor of shape [B, N, U, W, F_span]. batch_size (B), num_heads (N), num_query_blocks (U), query_block_size (W), key_context_size (C = W+L+R), max_span_plus_1 (F_span = L+R+1). Returns: Tensor of shape [B, N, U, W, C]. r"rN)rO functionalpadreshape)rTrrrwrrrrpad_amount_last_dim padding_tupleterm_bd_paddedterm_bd_reshapedterm_bd_slicedterm_bd_shifteds r3_relative_shiftz5Gemma3nAudioRelativePositionEmbedding._relative_shifts4 0!3F/0 **+?O *11  $4q$89   *!Q3X5EHX5X3X*XY)00     r2querieskeysc R|j\}}}}}|j\}}} }}tj|j|j dz d|j j d} | jd} |j| |j} |j| } | jd| |j|jjd}|jddddd}|jddddd}tj||}|jddddd}|jddd}|j||||z|}tj||}|j||||| }|j!|||||| | }||zS) Nr"rWrrrrr )rfr.rr}rrrrrrrrwrzsqueezepermutematmulr)rTrrrrrrwrz_r pos_indicesrsin_emb_timing_signalprojected_sin_embsin_emb queries_pkeys_p_tterm_ac q_permuted s_permuted q_reshapedterm_bd_unshifed_matmulterm_bd_unshifedrs r3rdz-Gemma3nAudioRelativePositionEmbedding.forwards OVmmK $&6 8'+zz$11ll4#4#4t7G7G6G!6KRX_XfXfgqq  &++A. $ > > w}}!?!  !MM*?@#++APTP]P]^ff  OOAq!Q2 <<1aA.,,y(3__Q1a3 __Q1-  '' I?ORb?bdlm #(,,z:"F3::       ..        ((r2) r*r+r,r#rNr.rkrrrirrdrlrms@r3roros 1 .)%,,)u{{)W\WcWc) ;#ll;; ;  ;  ;;; ;zL)u||L)5<<L)ELLL)r2roc*eZdZdeffd ZdZdejdededejfdZ d ejdejfd Z d ejdejfd Z d ejd ejdejfd Z xZS)Gemma3nAudioAttentionrpct|||_|jj|_|jj |_|j |jz|_|jj|_|jj|_ td|jjdz |_ |jj|_|j|jz|jz|_t#||_t'j(t+j,|j f|_t'j0|j |j|j zd|_t'j0|j |j|j zd|_t'j0|j |j|j zd|_|j dz}dt*j&j8j;t+j<dz }|j?d||zjAjCd |jE}|j?d |d |j?d t+j<|jjGd y) Nrr"FrrrJq_scalerKlocal_causal_valid_masksoftcap)$rMrNrprvrwrxrzconf_attention_chunk_size chunk_sizer~max_future_horizonr{r|max_past_horizonconf_attention_logit_capattention_logits_soft_cap context_sizerorelative_position_embeddingrOrPr.zeros per_dim_scalerq_projk_projv_projrsoftplusrSrRclonedetachcreate_local_causal_valid_maskra)rTrpr r_softplus_0rrUs r3rNzGemma3nAudioAttention.__init__Cs)  ==;;22((DNN: ++??"&++"J"J #At{{'N'NQR'R S)-)M)M& OOd.C.CCdF]F]]+PQW+X(\\%++t}}6F*GHii 0 0$..4==2PW\] ii 0 0$..4==2PW\] ii 0 0$..4==2PW\] --%UXX0099%,,s:KLL  Y<)?(F(F(H(O(O(Q^cd"&"E"E"G 68O\ab   LL77 8 > > @  r2ctjtj|j|jftj dj }tjtj|j|jftj |j|jz}tj|j|jftj }||z|z}|S)Nrr)diagonal) r.trilrQrrrjTrr)rTlower_causal_maskupper_causal_maskrs r3rz4Gemma3nAudioAttention.create_local_causal_valid_maskes!JJ JJ))4??;5:: N  ! "JJ JJ):):;5:: N**T-D-DD #(**doot?P?P-QY^YcYc"d"9}tjjjA|dtjB j/|j< }|j\}}}}}|jd}|jEd ddddj d||}|jEd ddddj d||}tjF||} | j |||||jEd dddd}!|!j | ||jHz|j|jf}!|!ddd| f}!|!S)NrWr"r rrz%Shape of extracted_valid_mask_blocks z is not (z, z) after potential reshape.rrErr)%rfrwrzrrrrrr.rOrrrviewrrrrr ValueErrorrr logical_andrrrrtanhwherefinforminsoftmaxrrbmmr)"rTr@r qkv_shape query_states key_states value_statesper_dim_scale_spbroadcast_shapeper_dim_scale_sp_broadcastrq_time query_blocks key_blocks value_blocksroriginal_valid_maskextracted_valid_mask_blockscondition_from_input_validitycondition_from_causalityfinal_condition_for_wherer> softcap_val probabilitiesb_dimn_dimu_dimw_dimc_dimh_dimprob_bunv_bun result_bmmcontext_vectorss" r3rdzGemma3nAudioAttention.forwardsTNm))#2.NN N {{=199)DOOQ [[/77 BMMO {{=199)DOOQ  88..778J8JKaDMM2%5%:%:?%K"#dll25OO )//3 F--l; 00< 22<@ '--a0 $e'+&A&ABU&V# ( , , 1+11!47R7X7XYZ7[[_c_p_pp*E*M*M,d.?.?+ ' ' , ,     1  /556i |L$%R(9(9'::TV )D(M(Ma(P(Z(Z[](^%$(#?#?#I#I!#L#V#VWX#Y#c#cde#f %*$5$5 ) $ ' '(E(L(L M% ! 11, Klloofmm4 +%F#+%6 FLL@Y@]@]^++33F%--3X[[bnbtbt[u -:,?,?)ueUE""2& ((Aq!Q7??E5Q$$Q1a3;;BuMYYx/ $,,UE5%OWWXY[\^_abdef)11 4??2   *!WfW*5r2)r*r+r,r#rNrr.rkrirrrr/rdrlrms@r3rrBs  1  D '5<<335<<u|| ,.'ELL.'U\\.'`dU\\d9I9Idelldr2rcreZdZdZ d dedeedeffd ZdejdejfdZ xZ S) Gemma3nAudioCumulativeGroupNormaApplies Group Normalization cumulatively over the time dimension. This layer normalizes the input by calculating the mean and variance cumulatively over the time dimension (dim 1). The statistics are computed over all feature dimensions (specified by `feature_dims` and `num_channels`) for elements marked as valid by the optional `mask`. If a `mask` is provided (True for valid, False for invalid/padded), invalid time steps do not contribute to the statistics calculation, and their corresponding output values are zeroed out. Scale and bias, if enabled, are applied per-channel (last dimension). This behavior is similar to JAX's `GroupNormalization` with `num_groups=1` and `cumulative=True`. num_channels feature_dimsrFc t|||_t||_||_t jtj||_ ttddt|jzdz|_ y)Nr r")rMrNr*rBr+rFrOrPr.rQrIrangelenreduction_axes)rTr*r+rFrUs r3rNz(Gemma3nAudioCumulativeGroupNorm.__init__7sr (!,/ll5::l#;< $E!QT5F5F1G-G!-K$LMr2r@r^c|j|jfz}|jdd|k7rtd|jddd||j}t j }|j|}t j||}t j||jd}t j|d }t j||jd} t j| d } t j| d } || z } || z jd} t j| |jd}t j|d }|| z }|| z t j||jzz}|j j|}dg|j#dz z|jgz}||j%|z}||z}|j|S) zApplies cumulative group norm, optionally using a mask. Args: hidden_states: Input tensor, shape [B, T, *feature_dims, C]. Returns: Normalized tensor with the same shape as x. r NzInput tensor shape suffix z> does not match expected suffix (feature_dims + num_channels) rTrErXr"rrJ)r )r+r*rfrrr.rr ones_likesumr/cumsumclamprZrsqrtrFrIrEr)rTr@expected_input_suffix input_dtype calc_dtypex_calc mask_calcsum_values_at_tcum_sum_valueselements_in_group_at_tcum_count_elementssafe_cum_count_elementscum_meansquared_diff_from_meansum_sq_diff_at_tcum_sum_sq_diff cum_variance normalized_xscalescale_view_shape final_outputs r3rdz'Gemma3nAudioCumulativeGroupNorm.forwardIs!% 1 1T5F5F4H H   qr "&; ;,]-@-@-D,EF99N8OQ  $)) ]] !!*- OOF*=  ))F0C0CTRo1="'9$:M:MW[!\"\\*@aH"'++.@c"J"$;; #)8"3!8!8!; 99%;ATAT^bc ,,'7Q?')@@ )U[[9P-QQ  z*3-"3"3"5"9:d>O>O=PP#ejj1A&BB $i/ {++r2)gMbP?) r*r+r,r-rirrarNr.rkrdrlrms@r3r)r)&sT( NNsmN N$G,U\\G,ellG,r2r)c ~eZdZdZ d dedededeeeeefffd Zdejdejfd Z xZ S) Gemma3nAudioSSCPConvBlockzA single convolution block for the SubSampleConvProjection. This block consists of a 2D convolution, followed by CumulativeGroupNorm, and a ReLU activation. It handles manual padding for the convolution. rpidxinput_freq_dimmanual_paddingcJt |||_||_|dk(rdn|jj|dz }|jj|}|jj |\}}|jj |\} } tj||||f| | fdd|_ ||jdz|jdz} | |z | zdz} t|| f|jj|_ tj|_y)Nrr")rrF) in_channels out_channels kernel_sizestridepaddingrs)r*r+rF)rMrNrprNsscp_conv_channel_sizesscp_conv_kernel_sizesscp_conv_stride_sizerOConv2dconvr)sscp_conv_group_norm_epsnormReLU activation)rTrprLrMrNrPrQkernel_hkernel_wstride_hstride_w f_in_padded f_out_convrUs r3rNz"Gemma3nAudioSSCPConvBlock.__init__s%  ,!8a)K)KCRSG)T {{99#> ![[>>sC(![[>>sC(II#%h'   %t':':1'==@S@STU@VV !H,9A= 3%$ 44  '')r2audio_encodingsr^ctj||jddj|jj j }|j |}|jddddj}|j|}|jddddj}|j|S)Nconstantr)modevaluerr rr") FrrNrrYrIrrrr[r])rTrdaudio_encodings_paddedaudio_encodings_conv x_for_normx_normedaudio_encodings_normeds r3rdz!Gemma3nAudioSSCPConvBlock.forwards"#8K8KR\dg!h!k!k II   " ""  $yy)?@*11!Q1=HHJ 99Z(!)!1!1!Q1!=!H!H!J566r2))rrrr) r*r+r,r-r#rirBrNr.rkrdrlrms@r3rKrKsc5A )$")$)$ )$ c3S01 )$V7u||7 7r2rKc\eZdZdeffd Zdej dej fdZxZS)#Gemma3nAudioSubSampleConvProjectionrpcpt|||_|j}g}g}t dD]n}|j |\}}|j |\}} d} |dz } d} d} | | | | f}|j||| z| z}||z | zdz}|j||}ptd|j||d|_ td|d||d|_ |jd}|d}||z|_ tj|j|jjd|_y)Nr rr")rLrMrprNrWFrr)rMrNrpinput_feat_sizer-rVrWappendrKconv_0conv_1rUinput_proj_in_featuresrOrrxinput_proj_linear)rTrpcurrent_f_for_block_inputcalculated_block_paddingcalculated_f_out_dimsir^r_r`ra pad_t_top pad_t_bottom pad_f_left pad_f_rightmanual_padding_tuplerbf_out_after_conv final_c_out final_f_outrUs r3rNz,Gemma3nAudioSubSampleConvProjection.__init__s  $*$:$:!#% "q 9A!'!=!=a!@ Hh!'!=!=a!@ HhI#a#R %=$B$B!1i#;#C#CAq)V^J^#_ ))$<=++o8N8N7NPTPfPfg,t~~o/NNNr2 r*r+r,r#rNr.rkr/rdrlrms@r3rr!sAA1AOu||OUEUEUOZ_ZfZfOr2rc\eZdZdeffd Zdej dej fdZxZS) Gemma3nAudioConformerFeedForwardrpct|||_|jdt j |jj dt|jj|_ tj|jj|jjdzd|_ tj|jjdz|jjd|_ t|jj|_|jj|_y)NrFrKrrr)rMrNrprRr.rSrrDrxpre_layer_normrOr ffw_layer_1 ffw_layer_2post_layer_normconf_residual_weightpost_layer_scalers r3rNz)Gemma3nAudioConformerFeedForward.__init__>s   0%,,t{{?\?\2]jop,T[[-D-DE99T[[%<%U>UXY>Y`ef99T[[%<%(,(8(8(I--,,_=(,(8(8(I++o8N8N7NPTPfPfg..??T-B-BBCCr2rrms@r3rr=s0 A1 A Du|| D Dr2rc\eZdZdeffd Zdej dej fdZxZS) Gemma3nAudioConformerLightConv1drpc t|||_t|jj|jj |_tj|jj|jjdzd|_ tj|jj|jj|jjdd|jjd|_ |jdtj|jj d t|jj|jj |_tj|jj|jjd|_|jjdz |_y) NrFr Frrr"r)rPrQrRrSrTgroupsrsrrK)rMrNrprDrx rms_norm_epsrrOr linear_startConv1dconf_conv_kernel_sizedepthwise_conv1drRr.rSr conv_norm linear_endcausal_paddingrs r3rNz)Gemma3nAudioConformerLightConv1d.__init__WsD  ,T[[-D-D$++JbJbcIIdkk&=&=t{{?V?VYZ?Zafg "  //00 99;;**!  0%,,t{{?\?\2]jop' (?(?T[[E]E]^))DKK$;$;T[[=T=T[`a"kk??!Cr2rdr^c:|}|j|}|j|}tjjj |d}|j ddd}tj||jdf}|j|}|j ddd}tj||j |j}|j|}tjj|}|j|}||z}|S)NrWrrr r")rrr.rOrglurrirrrr5rrrr)rTrdaudio_encodings_residualaudio_encodings_permutedaudio_encodings_permuted_paddedrcs r3rdz(Gemma3nAudioConformerLightConv1d.forwardls#2 --o>++O<((--11/r1J#2#:#:1a#C *+%%0H4K^K^`aJb*c'//0OP)11!Q:++o8N8N7NPTPfPfg..9--,,_=///: #;; r2rrms@r3rrVs-D1D*u|| r2rcteZdZdeffd Zdej dejdej fdZxZ S)Gemma3nAudioConformerBlockrpct|||_t|j|_t |j|_t|j|_t|j|_ |jdtj|jjdt|jj|_y)NrFrK)rMrNrprffw_layer_startr attentionrlconv1d ffw_layer_endrRr.rSrrDrxr[rs r3rNz#Gemma3nAudioConformerBlock.__init__s  ? L7 D7 D =dkkJ 0%,,t{{?\?\2]jop"4;;#:#:; r2rdr)r^cj|j|}|j||}|}||jdj|jz}|j |}|j |}tj||j |j}|j|}|S)NrW) rrrrrrrr.r5rr[)rTrdr)validity_mask_for_lconvaudio_encodings_for_lconv_inputrcs r3rdz"Gemma3nAudioConformerBlock.forwards..?...I#1/*9v?\?\9] ^A ' / ^   _sA=r)kwargsr^c >|j|}|jd}d}tt|jj D]!}||jj |dz}#t j||j|z}t j||jddz }|jdkDr>|jdk(r/|jdj|jdd}n`|j|jk(rG|jddk(r5|jddk7r#||jdk(r|jd}t j|d|} |jD] } | || } |jjdkDr@|dddd|jjf}| dddd|jjf} |j!| jdd}t#|| S) aEncodes a batch of MELs. Args: audio_mel: a torch.Tensor of shape [batch, num_frames, num_channels, mel_bins]. Returns: audio_encodings: a torch.Tensor of shape `[batch_size, self.config.audio_soft_tokens_per_image, self.config.audio_config.hidden_size]` audio_mel_mask: a torch.BoolTensor of shape [batch, num_frames]. r"rr)r{rWNr)last_hidden_stater))rrfr-r.rprWr.rrr5rrexpandgatherrconf_reduction_factor masked_fillr() rTrr)rrdt_subtime_stride_productstride_pair_idxindices current_maskblocks r3rdzGemma3nAudioEncoder.forwards 88C %%a($S)J)J%KL YO 4;;#D#D_#UVW#X X  Y ,,u^-B-BCFYY++g>+?+?+BQ+FG    "w||q'8''*11.2F2Fq2I2NG   7<< /$$Q'1, a A%q))''*G||NAw? ^^ CE#O\BO C ;; , ,q 0-a1UDKK4U4U1U.UVO'+Odkk.O.O+O(OPL)55l6L6LR6PRUV--'  r2)r*r+r,r-r#r0main_input_nameinput_modalitiesrNrr.rkr/rrrBr(rdrlrms@r3rrsr !O18 8 7<7G7G8 SYZlSm8 / /8 8 r2rc ZeZdZdZd dedededeffd Zdejffd Z xZ S) Gemma3nTextScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. num_embeddings embedding_dim padding_idx embed_scalect||||||_|jdt j |dy)NrFrK)rMrNscalar_embed_scalerRr.rS)rTrrrrrUs r3rNz'Gemma3nTextScaledWordEmbedding.__init__s;  D"- ]ELL,ERWXr2 input_idsct|||jj|jj zSr`)rMrdrrrIr)rTrrUs r3rdz&Gemma3nTextScaledWordEmbedding.forwards2wy)D,<,<,?,? @Q@Q,RRRr2)rJ) r*r+r,r-rirarNr.rkrdrlrms@r3rrsGYsY3YSY_dY SSSr2rc`eZdZdZdeffd ZdejdejfdZxZ S)Gemma3nTextLaurelBlockz Learned Augmented Residual Layerrpct|||_tj|jj |jj d|_tj|jj |jj d|_t|jj |jj|_ y)NFrrr) rMrNrprOrrx laurel_rank linear_left linear_rightrDrpost_laurel_normrs r3rNzGemma3nTextLaurelBlock.__init__s  99T[[%<%U>U\abIIdkk&=&=t{{?V?V]bc .t{{/F/FDKKLdLd er2r@r^cr|j|}|j|}|j|}||zSr`)rrr)rTr@laurel_hidden_statesnormed_laurel_hidden_statess r3rdzGemma3nTextLaurelBlock.forwardsC-1-=-=m-L-1->->?S-T&*&;&; YYv2teL n!<<}}!!&8"344r2)r) r*r+r,r%rirNr.rkrdrrlrms@r3rr sP Q0 QS QU\\ell5U\\5ell5r2rcXeZdZdZdeffd ZdejdejfdZdejdejfdZ d ejd ejdejfd Z d ejdejfd Z d ejdejfdZ xZ S)Gemma3nTextAltUpaAlternating Updates (AltUp) The AltUp module wraps transformer layers. The `predict` step modifies the input to the transformer layer, and the `correct` step propagates the output of the transformer layer to the sparsely updated dimensions. See more in the research paper: https://proceedings.neurips.cc/paper_files/paper/2023/file/f2059277ac6ce66e7e5543001afa8bb5-Paper-Conference.pdf rpcFt|||_tjt j |jj|_tj|jj|jjd|_ tj|jj|jjdzd|_ tj|jj|jjd|_ t|jj|jj|_|j#dt j$|jjdzdy)NFrrr rrouter_input_scalerK)rMrNrprOrPr.rrxcorrect_output_scaleraltup_num_inputscorrection_coefsprediction_coefsmodality_routerrDr router_normrRrSrs r3rNzGemma3nTextAltUp.__init__>s  $&LLT[[=T=T1U$V! " $++*F*F HdHdkp q " $++*F*F HdHdfgHgns t!yy)@)@$++B^B^ejk)$++*A*At{{G_G_` 15<< @W@WY]@]3^kpqr2r\r^c|j||jz}|j|}tj|j j |Sr`)rrrr.rrarb)rTr\ router_inputsrouteds r3compute_router_modalitiesz*Gemma3nTextAltUp.compute_router_modalitiesHsM((+d.E.EE %%m4zz&,,.)11!44r2r@c|j||jj}|jro|jjY|j j jj|jj |jj|j |jg|jdd|jj|jjjdddd}tj|jdddd|}|jdddd}||z }|jj!|S)aPredicts the output of a layer using a trainable map. Args: hidden_states: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` derived by stacking the input embeddings and preprocessing the last `num_altup_inputs - 1` matrices. Returns: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` containing the predictions. NrWrr"rr )rrpaltup_active_idxtrainingaltup_coef_cliprrIdataclamp_rrfrrr.rrrb)rTr@ modalities all_coefs predictionss r3predictzGemma3nTextAltUp.predictMs@33M$++B^B^4_` ==T[[88D  ! ! ( ( - - 4 4dkk6Q6Q5QSWS^S^SnSn o D ! !* - W i &&s+ i-1[[-I-I iKO;;KgKg i WQ1a  ll=#8#8Aq!#DiP !))!Q15 }$ %%'// >>r2r( activatedc|j|}|||jjz }|j|jjddd}|j ro|jj Y|jjjj|jj |jj |j|dz}|jdddjd}tj||}||z }|jj!|S)a_Corrects the predictions relative to the Args: predictions: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` derived by stacking the input embeddings and preprocessing the last `num_altup_inputs - 1` matrices. activated: A 3D tensor of shape `[batch_size, num_tokens, hidden_size]` containing the activated inputs. Returns: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` correcting the original predictions relative to the activated input embeddings. r"rJr rrW)rrpr!repeatrr"r#rrIr$r%rrr.mulrrb)rTr(r*r& innovationr' correcteds r3correctzGemma3nTextAltUp.correctis 33I> T[[-I-I!JJ &&t{{'C'CQ1M ==T[[88D  ! ! ( ( - - 4 4dkk6Q6Q5QSWS^S^SnSn o #'"7"7 "Cc"I %%aA.88< IIj)4 [ ##%--i88r2r/cp|j|j|jzj|S)a  This is only defined as the `forward` so that accelerate hooks can move correctly `correct_output_scale` (which is a nn.Parameter, not a Module) between devices when offloading. It is otherwise only used in `scale_corrected_output` )rbrrTr/s r3rdzGemma3nTextAltUp.forwards2 !!$";";