i5ddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z dd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZddlmZmZddlmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.ddl/m0Z0e*jbe2Z3edGddejhZ5GddejhZ6GddejhZ7GddejhZ8Gdd ejhZ9Gd!d"ejhZ:d#Z;ed$dCd%Zd(ejzfd)Z? dDd*ejhd+ejzd,ejzd-ejzd.ejzdzd/e@d0e@d1e%e'fd2ZAGd3d4ejhZBGd5d6eZCe(Gd7d8e#ZDe(Gd9d:eDZE dEd;ejzeFejzzdzddzd.ejzdzd(ejze>zfd=ZGGd>d?eDeZHGd@dAeeDZIgdBZJy)F)Callable)OptionalN)nn) functional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hub)create_causal_mask) GenericForSequenceClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)OutputRecordercheck_model_inputsmaybe_autocast) JetMoeConfigRMSNormc,eZdZdfd ZdZdZxZS) JetMoeRMSNormct|tjt j ||_||_y)z< JetMoeRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s t/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/jetmoe/modeling_jetmoe.pyr'zJetMoeRMSNorm.__init__1s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor)float32powmeanrsqrtr,r+)r- hidden_states input_dtypevariances r1forwardzJetMoeRMSNorm.forward9sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r2c^t|jjd|jS)Nz, eps=)tupler+shaper,)r-s r1 extra_reprzJetMoeRMSNorm.extra_repr@s* ))*+6$2G2G1HIIr2)gư>)__name__ __module__ __qualname__r'r@rD __classcell__r0s@r1r$r$/s$;Jr2r$c eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)JetMoeRotaryEmbeddinginv_freqNconfigct||j|_|j|_||_|j j d|_|j}|jdk7rt|j}||j |\}|_ |jd|d|jd|jdy)N rope_typedefaultrLF) persistentoriginal_inv_freq) r&r'max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrMrope_parametersrOcompute_default_rope_parametersrattention_scalingregister_bufferclone)r-rMdevice rope_init_fnrLr0s r1r'zJetMoeRotaryEmbedding.__init__Gs "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuUr2r[z torch.deviceseq_lenreturnz torch.Tensorc|jd}t|ddxs|j|jz}d}d|t j d|dtj j|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr4r7)r[r7) rVgetattrr.num_attention_headsr)arangeint64r8float)rMr[r]basedimattention_factorrLs r1rWz5JetMoeRotaryEmbedding.compute_default_rope_parametersWs&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k )))r2cN|jddddfjj|jdddj |j }|dddddfj}t |j jtr/|j jdk7r|j jnd}t|d5|j|jzjdd}tj||fd }|j|jz}|j|jz} dddj |j  j |j fS#1swYAxYw) Nrr5r mpscpuF) device_typeenabledr4rirb)rLrgexpandrCr8r[ isinstancetypestrr transposer)catcosrXsinr7) r-x position_idsinv_freq_expandedposition_ids_expandedrnfreqsembrwrxs r1r@zJetMoeRotaryEmbedding.forwardusR!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk  U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BFF$N)NNN)rErFrGr)Tensor__annotations__r!r' staticmethodrintrBrgrWno_gradrr@rHrIs@r1rKrKDsllV|V &*+/"*t#*(*t* ~u$ % **:U]]_ < (=a(@ADYZ[D\\ ++M;G'+ag*>> S6\4??;>CWCW`n`u`uvq+~F #((fdooF #dii/ r2)rErFrG__doc__r!r'r@rHrIs@r1rrs | r2rc:eZdZdZdeffd ZdZdZdZxZ S) JetMoeMoAz A Sparsely gated mixture of attention layer with pairs of query- and output-projections as experts. Args: config: Configuration object with model hyperparameters. rMcht||j|_|j|_|j |jz|_|j|_ tjjtj|j |_t|j|j |j|_t|j|j|j |_t%|j |j|j|_y)Nr)r&r'rrr.r kv_channelsnum_key_value_headsrrr)rr(rrrrrrrrs r1r'zJetMoeMoA.__init__&s !33 ,,!--0J0JJ// HH&&u{{4??'CD 1$2B2BDOOUYUeUef243C3CTEUEUW[WfWfg&((**  r2c|j\}}}|jd|}|j|\}}}}} ||||f} ||} |j| |} t j ||z|j z|jf| j| j} | jd|| }|j|||j d}|| | fS)z Map inputs to attention experts according to routing decision and compute query projection inside each experts. r5rr) rrrrr)rrr.r7r[rr)r-rrrrrrrrr topo_inforrrrs r1mapz JetMoeMoA.map8s !, 0 0 2VX!))"h7 UYU`U`alUmRk; ]); [Q $K0 **=+F 6\DJJ &(8(8 9AUAU^l^s^s q*>O #((fdjj"E ]I55r2c|j\}}}}|jd|}|\}}} } ||} |j| | } | | dddfz} tj||z|j f| j | j} | jd|| }|j|||j }||jz}|S)zu Compute output projection inside each attention experts and merge the outputs of different experts. r5Nrr) rrrr)rrr7r[rrr)r-rrrrkr.rrrrrrrrs r1reducezJetMoeMoA.reduceOs'2&6&6&8#VQ !))"k: FOCk; $$89 ++M;G(+ag*>> S6\4??;>CWCW`n`u`uvq+~F #((fdooF #dii/ r2ctd)Nz-This module doesn't support call and forward.)NotImplementedError)r-rs r1r@zJetMoeMoA.forwardes!"QRRr2) rErFrGrr!r'rrr@rHrIs@r1rrs$ | $6.,Sr2rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr5r4rp)rCr)rv)ryx1x2s r1 rotate_halfrisZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r2rotary_pos_embc|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qrrwrx unsqueeze_dimq_embedk_embeds r1apply_rotary_pos_embrpsY& -- &C -- &C3w;q>C/0G3w;q>C/0G G r2r=n_repr^c|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r N)rCrqr)r=rbatchrslenras r1 repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr2modulequerykeyvalueattention_maskscalingdropoutkwargsc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr4rr5)rir7)ptrainingr )rnum_key_value_groupsr)matmulrurCrrrr9r8r7rr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r1eager_attention_forwardr s3 ; ;#D#DQB#D#O L#((6@@AF __\2<E+, r2r'ceZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eej dd eeed d Zej*fd ZxZS) JetMoePreTrainedModelrMmodelFr'rTgater ) layer_nameindex)r7)rr= attentionsc t||t|tr7t j |j d|jjyt|ttzr t j|jyy)zInitialize the weights.r)r;stdN) r& _init_weightsrrrinitnormal_r+rMinitializer_rangerrzeros_r)r-rr0s r1r;z#JetMoePreTrainedModel._init_weightsHs_ f% f3 4 LLSdkk6S6S T  I 5 6 KK $7r2)rErFrGr!rbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrrrr'r _can_record_outputsr)rr;rHrIs@r1r3r36s &+#-.#4"5N""&' fAN+$_A> U]]_%%r2r3ceZdZdeffd Zee ddejdzdejdzdejdzde dzdejdzd e dzd ejdzd e ed efd ZxZS) JetMoeModelrMc .t|||j|_|j|_t j |j|j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_t#||_d|_|j(|_|j+ycc}w)N)r/rMF)r&r' pad_token_id padding_idx vocab_sizer Embeddingr. embed_tokens ModuleListrnum_hidden_layersr'layersr$ rms_norm_epsnormrK rotary_embgradient_checkpointingr post_initrs r1r'zJetMoeModel.__init__Ts  !.. ++LL):):F&+#$*$?$?!  esDN input_idsrrzr inputs_embedsr-rrr^c D|du|duz r td|r|t|j}||j|}|F||j nd} t j | | |jdz|j}||jd}t|j|||||} |} |j| |} |jd|jjD]} | | f| | ||||d|} |j| } t| |S) Nz:You must specify exactly one of input_ids or inputs_embedsrMrr )r[)rM input_embedsrrrrz)rrrr-rrz)last_hidden_stater) ValueErrorr rMrRget_seq_lengthr)rerCr[rrrXrUrTrWr)r-r[rrzrr\r-rrpast_seen_tokensrr=r decoder_layers r1r@zJetMoeModel.forwardes`  -t"; <YZ Z 0*$++>O   --i8M  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L(;;&))+%  & #oom\J![[)H4;;+H+HI M) $7* /#-)  M  -0 %++  r2)NNNNNNN)rErFrGr!r'rrr)r%rr FloatTensorr1rrrr@rHrIs@r1rKrKRs|".2.204(,26!%26; ##d*;  t+; &&- ;  ; ((4/ ; $;; ((4/; +,;  ; ; r2rK gate_logitsrc|t|tsyt|trC|dj}tj|Dcgc]}|j |c}d}tj jjd}tj||d\}} tj jj| |} |>tj| jd} tj|d} n|j\} }|jd| |zz}|dddddddfj|| |||fjd||j }tj | j|zdtj |dz } |ddddddfj|| ||fjd|j |}tj ||zdtj |dz } tj | | j#dz}||zScc}w)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrrpr5)rrrBr[r)rvr8rrrrone_hotr;rgrCrqrrr)rerrrcompute_device layer_gateconcatenated_gate_logitsrouting_weightsrselected_experts expert_masktokens_per_expertrouter_prob_per_expert batch_sizesequence_lengthrTexpert_attention_mask router_per_expert_attention_mask overall_losss r1load_balancing_loss_funcrus9:*[%"@+u%$Q..#(99^i-jPZjmmN.K-jpq#r hh))112JPR1SO**_eDA((%%--.> LK!JJ{'8'8':B"'O!C&4&:&:# O4::1=*B^_ 4AtT1 2 V& OUKX Y WR , R  "IIk&7&7&9SV8W~ot4]k mA}a,?@A  %4%% ;;11 D /%%  (( H !**X[[-EEE(#33!//))!//  r2) NNNNNNNNrF)rErFrG_tied_weights_keysr'rrr)r%rr rdr1rrr@rHrIs@r1rwrws*,GH .2.204(,26*.!%26-.,1: ##d*:  t+: &&- :  : ((4/ :   4': $;: ((4/: ell*: #Tk:  #: : r2rwc eZdZy)JetMoeForSequenceClassificationN)rErFrGr/r2r1rrDsr2r)rwrKr3r)r )r)Nr4N)Kcollections.abcrtypingrr)rtorch.nnrrrr< activationsr cache_utilsr r generationr integrationsr r masking_utilsrmodeling_layersrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrrconfiguration_jetmoer! get_loggerrErModuler$rKrrrrrrrrrrgr r r'r3rKrBrurwr__all__r/r2r1rs*% $&!.)Q/[QKF&RROO.   H %Y'JBIIJ(J(>