i^dZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l mZmZdd lmZdd lmZdd lmZdd lmZmZddlmZmZddlmZddlmZm Z m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1e"jde3Z4Gdde+Z5Gdde,Z6GddejnZ8GddejnZ9Gdd ejnZ:Gd!d"ejnZ;Gd#d$ejnZ<Gd%d&e'Z=e Gd'd(e*Z>e Gd)d*e)Z?Gd+d,e>eZ@Gd-d.ee>ZAgd/ZBy)0zPyTorch JetMoe model.)CallableN)nn) functional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask) GenericForSequenceClassification)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)OutputRecordercheck_model_inputs)LlamaDecoderLayer) MixtralModelMixtralPreTrainedModelMixtralRMSNormMixtralRotaryEmbeddingapply_rotary_pos_embeager_attention_forwardload_balancing_loss_func) JetMoeConfigc eZdZy) JetMoeRMSNormN__name__ __module__ __qualname__s/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/jetmoe/modular_jetmoe.pyr%r%3r+r%c eZdZy)JetMoeRotaryEmbeddingNr&r*r+r,r/r/7r-r+r/c6eZdZdedededdffd ZdZxZS)JetMoeParallelExperts num_experts input_size output_sizereturnNct|tjt j ||||_||_||_||_ y)a Initialize the JetMoeParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) super__init__r Parametertorchemptyweightr2r3r4)selfr2r3r4 __class__s r,r8zJetMoeParallelExperts.__init__<sD" ll5;;{K#TU &$&r+c|j|d}g}t|jD]7}|jt j |||j |9tj|d}|S)a Forward pass of the JetMoeParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rdim) splitranger2appendFlinearr<r:cat)r=inputs expert_size input_list output_listiresultss r,forwardzJetMoeParallelExperts.forwardSsq\\+1\5  t''( HA   qxx 1 t{{1~F G H))KQ/r+r'r(r)intr8rN __classcell__r>s@r,r1r1;s)'C'S's't'.r+r1c2eZdZdededeffd ZdZxZS)JetMoeTopKGatingr3r2top_kct|||_||_||_t j ||d|_y)a Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. FbiasN)r7r8r2r3rUrLinearlayer)r=r3r2rUr>s r,r8zJetMoeTopKGating.__init__is: &$ YYz;UC r+c|j|j}|j|jd\}}t j |dj |}t j|jd|jg|j|j}|jd|d}|jjd}|j}|j!} | j#d\} } | j%|jd} |j!}|| } | | | ||fS)Nr"r@rdtypedevicetrunc) rounding_mode)rZfloattopkrUr:softmaxtype_aszerossizer2r]r^scatterlongsumtolistflattensortdiv)r= hidden_stateslogits top_k_logits top_k_indices top_k_gatesregatesrI top_k_experts_index_sorted_experts batch_index batch_gatess r,rNzJetMoeTopKGating.forward}s.M*002&,kk$**!k&D# mmmLa8@@O    a $"2"2 3;;L;LU`UgUg  a2jjl&&q) "((* &--/ "/"4"4Q"7 *..tzz.Q "))+ !"67 #[+{FRRr+rOrRs@r,rTrThs'D3DSDD(Sr+rTc.eZdZdZdeffd ZdZxZS) JetMoeMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. configc@t||j|_|j|_t |j |_tjjtj|j|_ t|j|j|jdz|_t|j|j|j|_t#|j|j|j$|_y)Nrr3r2rU)r7r8 hidden_sizer3intermediate_sizeractivation_function activationr:rr9r;rXr1num_local_experts input_linear output_linearrTnum_experts_per_tokrouterr=r{r>s r,r8zJetMoeMoE.__init__s  ,,!33 !;!;<HH&&u{{4??'CD 1&2J2JDOO]a]m]mpq]qr263K3KTM]M]_c_n_no&00,,  r+c8|j\}}}|jd|}|j|\}}}}} ||} |j| |} | j dd} |j | d| dz} |j | |} | |dddfz} tj||z|jf| j| j}|jd|| }|j|||j}||jz}|S)a Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. rr@rr"Nr\)rfreshaperrchunkrrr:rer3r]r^ index_addviewrX)r= layer_inputbszlengthemb_sizerurwrxrI router_logits expert_inputsrnchunked_hidden_statesexpert_outputsre layer_outputs r,rNzJetMoeMoE.forwards%!, 0 0 2VX!))"h7 BF++kBZ?; [-#K0 ))-E - 3 3A2 3 >(=a(@ADYZ[D\\ ++M;G'+ag*>> S6\4??;>CWCW`n`u`uvq+~F #((fdooF #dii/ r+)r'r(r)__doc__r#r8rNrQrRs@r,rzrzs | r+rzc:eZdZdZdeffd ZdZdZdZxZ S) JetMoeMoAz A Sparsely gated mixture of attention layer with pairs of query- and output-projections as experts. Args: config: Configuration object with model hyperparameters. r{cht||j|_|j|_|j |jz|_|j|_ tjjtj|j |_t|j|j |j|_t|j|j|j |_t%|j |j|j|_y)Nr})r7r8rr2r~r3 kv_channelsnum_key_value_headsrrUr:rr9r;rXr1rrrTrrs r,r8zJetMoeMoA.__init__s !33 ,,!--0J0JJ// HH&&u{{4??'CD 1$2B2BDOOUYUeUef243C3CTEUEUW[WfWfg&((**  r+c|j\}}}|jd|}|j|\}}}}} ||||f} ||} |j| |} t j ||z|j z|jf| j| j} | jd|| }|j|||j d}|| | fS)z Map inputs to attention experts according to routing decision and compute query projection inside each experts. rr\r) rfrrrr:rerUr~r]r^rr)r=rrrrrvrwrxrIr topo_inforrrers r,mapz JetMoeMoA.maps !, 0 0 2VX!))"h7 UYU`U`alUmRk; ]); [Q $K0 **=+F 6\DJJ &(8(8 9AUAU^l^s^s q*>O #((fdjj"E ]I55r+c|j\}}}}|jd|}|\}}} } ||} |j| | } | | dddfz} tj||z|j f| j | j} | jd|| }|j|||j }||jz}|S)zu Compute output projection inside each attention experts and merge the outputs of different experts. rNr\r) rfrrr:rer3r]r^rrrX)r=rrrrkr~rvrwrxrIrrrers r,reducezJetMoeMoA.reduces'2&6&6&8#VQ !))"k: FOCk; $$89 ++M;G(+ag*>> S6\4??;>CWCW`n`u`uvq+~F #((fdooF #dii/ r+ctd)Nz-This module doesn't support call and forward.)NotImplementedError)r=rs r,rNzJetMoeMoA.forwards!"QRRr+) r'r(r)rr#r8rrrNrQrRs@r,rrs$ | $6.,Sr+rc eZdZdZd dededzffd Z ddejdejdzdejdzd e dzd ejdzd e ejejdze ejdzff d Z xZ S)JetMoeAttentionzH Multi-headed attention from 'Attention Is All You Need' paper. Nr{ layer_idxcbt|||_||_d|_|-t j d|jjdd|_ |j|_ |j|_ |j|jz|_|j|_|j |_|j|_|j$dz|_t)||_t,j.j1|j2|jdzd |_y) z Initialize the JetMoeAttention module. Args: config: Configuration object with model hyperparameters. layer_idx: Index of the layer in the model. TNzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r"grFrW)r7r8r{r is_causallogger warning_oncer>r'num_key_value_groupsrrUattention_dropoutrrkv_projection_sizenum_attention_heads num_headshead_dimscalingrexpertsr:rrYr~kv_projr=r{rr>s r,r8zJetMoeAttention.__init__$s  "      !8!8 9:,,  %&!// !'!9!9"("4"4v7Q7Q"Q#)#=#= 33** }}d*  ( xxv'9'94;R;RUV;V]bc r+rnattention_maskposition_embeddingspast_key_valuescache_positionr5c |jdd}g|d|j}|jj|\} } } |j |j dd\} } | j |jdd} | j |jdd} | j |jdd} |\}}t| | ||\} } |'|||d}|j| | |j|\} } tj|jjt}| j!d|j"dd} | j!d|j"dd} ||| | | |f|j$sdn |j&|j(d|\}}|j g||j"d}|jj+|| }|j g|d}||| fS)Nrrr@r")sincosr)dropoutr)shaperrrrrr transposerupdaterr get_interfacer{_attn_implementationr repeatrUtrainingrrr)r=rnrrrrkwargs input_shape hidden_shape query_statesrr key_states value_statesrr cache_kwargsattention_interface attn_output attn_weightss r,rNzJetMoeAttention.forwardEs$))#2. 88b8$--8 151A1A-1P. mY#'<< #>#D#DQB#D#O L#((6@@AF __\2<E+, r+rceZdZUeej ddeeeddZe e d<dZ dZ d gZ d gZd Zd ZdZej&d Zy )JetMoePreTrainedModelgater") layer_nameindex)r)rrn attentionsr{modelFrrTc,tj||t|tr7t j |j d|jjyt|ttzr t j|jyy)zInitialize the weights.r)meanstdN) r _init_weights isinstancer1initnormal_r<r{initializer_rangerrzzeros_rX)r=modules r,rz#JetMoePreTrainedModel._init_weightssa %%dF3 f3 4 LLSdkk6S6S T  I 5 6 KK $7r+N)r'r(r)rrrYrr_can_record_outputsr#__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraphr:no_gradrr*r+r,rrsz( fAN+$_A> &+#-.#4"5N"U]]_%%r+rceZdZdeffd Zee ddejdzdejdzdejdzde dzdejdzd e dzd ejdzd e ed efd ZxZS) JetMoeModelr{c t|||j|_|j|_t j |j|j|j|_t jt|jDcgc]}t||c}|_ |j|_t|j|j |_ycc}w)N)eps)r7r8 pad_token_id padding_idx vocab_sizer Embeddingr~ embed_tokens ModuleListrCnum_hidden_layersrlayersrr% rms_norm_epsnormrs r,r8zJetMoeModel.__init__s  !.. ++LL):):FO   --i8M  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L(;;&))+%  & #oom\J![[)H4;;+H+HI M) $7* /#-)  M  -0 %++  r+)NNNNNNN)r'r(r)r#r8rrr:rrr FloatTensorrrrrrNrQrRs@r,rrs O| O.2.204(,26!%26; ##d*;  t+; &&- ;  ; ((4/ ; $;; ((4/; +,;  ; ; r+rcLeZdZddiZfdZee ddejdzdejdzdejdzde dzd ejdzd ejdzd e dzd ejdzd e ejzde dzdefdZxZS)JetMoeForCausalLMzlm_head.weightzmodel.embed_tokens.weightc,t||t||_|j|_|j |_t j|j|jd|_ |j|_ |jy)NFrW) r7r8rrr aux_loss_coefrrYr~lm_headtie_word_embeddings post_initrs r,r8zJetMoeForCausalLM.__init__sq   (  ++#11yy!3!3V5F5FUS #)#=#=  r+Nr rrrr labelsrrlogits_to_keepoutput_router_logitsr5c L|jd|||||||d| } | j} t| tr t | dn| }|j | dd|ddf}d}|*|j ||fd|jji| }d}| rYt| j|j|j|}|+||j|j|jzz }t!|||| j"| j$| j&| jS)N)r rrrr rrr)lossaux_lossrorrnrrr*)rrrrPslicer loss_functionr{rr!rr2rrtor^rrrnr)r=r rrrr rrrr r!routputsrn slice_indicesror#r$s r,rNzJetMoeForCausalLM.forwardsQ +5$** + )%+') +  +  11 8B>SV8W~ot4]k mA}a,?@A  %4%% ;;11 D /%%  (( H !**X[[-EEE(#33!//))!//  r+) NNNNNNNNrF)r'r(r)_tied_weights_keysr8rrr:rrr rrrPrrNrQrRs@r,rr s*,GH .2.204(,26*.!%26-.,1: ##d*:  t+: &&- :  : ((4/ :   4': $;: ((4/: ell*: #Tk:  #: : r+rc eZdZy)JetMoeForSequenceClassificationNr&r*r+r,r,r,Zsr+r,)rrrr,)Crcollections.abcrr:rtorch.nnrrErr activationsr cache_utilsr r generationr masking_utilsr modeling_layersr modeling_outputsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrllama.modeling_llamarmixtral.modeling_mixtralrrrrrr r!configuration_jetmoer# get_loggerr'rr%r/Moduler1rTrzrrrrrrr,__all__r*r+r,r@sR$ $&!.)/RF&RR?4/   H % N  2 *BII*Z.Sryy.Sb7 7tIS ISXX8biiX8v(*(V%2%%2J ,J J ZJ -J Zd&FH]c kr+