iWvdZddlmZddlmZddlZddlmZddlmZ ddl m Z dd l m Z mZdd lmZdd lmZdd lmZdd lmZmZddlmZmZddlmZddlmZmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&ddl'm(Z(ddl)m*Z*Gdde$Z+GddejXZ-GddejXZ.GddejXZ/Gdd ejXZ0Gd!d"ejXZ1Gd#d$ejXZ2Gd%d&eZ3Gd'd(eZ4eGd)d*e4Z5Gd+d,e4eZ6gd-Z7y).z"Modular components for DBRX model.)Callable)AnyN)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask)GradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)check_model_inputs)LlamaRotaryEmbeddingapply_rotary_pos_embeager_attention_forward)load_balancing_loss_func) DbrxConfigc eZdZy)DbrxRotaryEmbeddingN)__name__ __module__ __qualname__o/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/dbrx/modular_dbrx.pyrr,sr$rceZdZdZ d dedzffd Z d dejdejdzdejdzde dzd ejdzd e ejejff d Z xZ S) DbrxAttentionzYModular DBRX attention component that can be reused across different model architectures.N layer_idxc t|||_|j|_|j |_|j|j z|_|j|_ ||_ |j}|j|_ |j|_|j|_|j |j z|_|jdz|_|j&|_d|_t+j,|j|jd|j z|jzzd|_t+j,|j|jd|_y)NgTrFbias)super__init__configd_model hidden_sizen_heads num_headshead_dim max_seq_lenmax_position_embeddingsr( attn_config attn_pdropattention_dropoutclip_qkv kv_n_headsnum_key_value_headsnum_key_value_groupsscaling rope_theta is_causalrLinearWqkvout_proj)selfr.r(kwargsr6 __class__s r%r-zDbrxAttention.__init__3s*  !>>((DNN: '-'9'9$"(( !,!7!7#,, #.#9#9 $(NNd6N6N$N!}}d* %00II   d..T5M5M1MPTP]P]1]]di   $"2"2D4D4D5Q r$ hidden_statesattention_maskposition_embeddingspast_key_valuescache_positionreturnc |jdd}g|d|j}|j|} |j |j nd} | j | |j} | j |j |j|jz|j|jzgd\} } } | j|jdd} | j|jdd} | j|jdd} |\}}t| | ||\} } |'|||d}|j| | |j|\} } tj|jj t"}||| | | |f|j$sdn |j&|j(d|\}}|j*g|dj-}|j/|}||fS) N)minmaxrdimr)sincosrJ)dropoutr=)shaper3rAr9clampsplitr0r;view transposerupdater(r get_interfacer._attn_implementationrtrainingr8r=reshape contiguousrB)rCrFrGrHrIrJrD input_shape hidden_shape qkv_statesmin_val query_states key_states value_statesrSrR cache_kwargsattention_interface attn_output attn_weightss r%forwardzDbrxAttention.forwardOs$))#2. 88b8$--8 YY}- $(MM$=4==.4%%'t}}%E 1;1A1A  ((4==8((4==8   2B2 . j,$((6@@AF __\2</37;(,26 6)||6) t+6)#--4 6)  6) ((4/ 6) u||U\\) *6)r$r'c eZdZfdZdej dej dej dej dej f dZxZS) DbrxExpertGLUct||j|_|j|_|j|_t j tj|j|jz|j|_ t j tj|j|jz|j|_ t j tj|j|jz|j|_ |jjdd}t||_y)Nnamesilu)r,r-r0ffn_hidden_sizemoe_num_expertsr Parameterrqemptyw1v1w2 ffn_act_fngetr activation_fn)rCr. act_fn_namerEs r%r-zDbrxExpertGLU.__init__s !--%55%55,,u{{4+?+?$BVBV+VX\XhXhij,,u{{4+?+?$BVBV+VX\XhXhij,,u{{4+?+?$BVBV+VX\XhXhij''++FF; #K0r$x expert_w1 expert_v1 expert_w2rKc|j|}|j|}|j|}||z}|j|j}|Srm)matmulrt) rCrrrr gate_projup_projintermediate_states down_projs r%rlzDbrxExpertGLU.forwardsWHHY' ((9%&&y1 ''1'..y{{}= r$r r!r"r-rqrrrlrurvs@r%rxrxsK 1*/,,CH<<\a\h\h r$rxceZdZfdZdej dej dej dej fdZxZS) DbrxExpertsct|t||_|j|_|j |_|j |_yrm)r,r-rxmlpr0r|r} num_expertsrCr.rEs r%r-zDbrxExperts.__init__sD  (!--%55!11r$rF top_k_index top_k_weightsrKc|jd}|jd|j}tj||j |j }tj5tjjj||j}|jddd}tj|jddj}dddd|j|j f}D]} | d} tj5tj"| \} } ddd|j$j&j)|| } |j$j*j)|| } |j$j,j)|| }|j%| | | |}|j)d|j||  dfz}|j/d| | |j)|d|j}|S#1swYOxYw#1swYxYw) NrrM)dtypedevice) num_classesrr)rMrP)rVr_r|rq zeros_likerrno_gradr functionalone_hotrpermutegreatersumnonzeror0whererrrYrr index_add_)rCrFrr batch_size next_states expert_mask expert_hitsplit_expert_shape expert_idxidx token_idxrrrstatess r%rlzDbrxExperts.forwards #((+ %--b$2F2FG &&}M>> !  ll6>>> r$rFrHrGrIrJrDrKc f|}|j|j|j}|jd|||||d|\}}tj j ||j|j}||z}|}|j|j|j}||fSN)rFrGrHrIrJ)rr^r#) rtorrrrrUrr^r) rCrFrHrGrIrJrDresidual_states_s r%rlzDbrxNormAttentionNorm.forwards( M255m6I6IJ $499 ') 3+)     q --mt?O?OZ^ZgZg-h %7 ' M255m6I6IJ  --r$rm)NNN)r r!r"rrpr-rqrrrsr rrtrlrurvs@r%rrs ?z ?cDj ?/3(,26 .||.#--. t+ .  . ((4/ .. u||U\\) *.r$rc eZdZdedeffd Z d dejdejdzdejdzde dzd ejdzd e f d Z xZ S) DbrxBlockr.r(ct||j|_|j|_||_t |||_t||_ y)Nrr.) r,r-r/r0rr(rnorm_attn_normrffnrs r%r-zDbrxBlock.__init__sP !>>!--"3 &)r$NrFrGrHrIrJrDc |jd|||||d|\}}|j|}tjj ||j |j }||z}|Sr)rrrrrUrr^)rCrFrGrHrIrJrD resid_statess r%rlzDbrxBlock.forward*s':d&9&9' ') 3+) '  ' # m/  --mt?O?OZ^ZgZg-h $}4 r$rn) r r!r"rrpr-rqrrrsr rrlrurvs@r%rrs *z *c */37;(,26 || t+#--4    ((4/ r$rceZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZej$dej(ffd ZxZS) DbrxPreTrainedModelr. transformerTrrIF)rF attentionsmodulec>t|||jj}t |t rgt j|jd|t j|jd|t j|jd|yy)NrT)meanstd) r, _init_weightsr.initializer_range isinstancerxinitnormal_rrr)rCrrrEs r%rz!DbrxPreTrainedModel._init_weightsRsj f%kk++ fm , LL# 6 LL# 6 LL# 6 -r$)r r!r"r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flex_attn_supports_attention_backend_supports_flash_attn_supports_sdpa_can_compile_fullgraphrr'_can_record_outputsrqrrModulerrurvs@r%rrBsx %&*#$ #4"5"&N""# U]]_7BII77r$rcLeZdZdZdeffd ZdejfdZdejfdZ e e dd e jdzd e jdzd e jdzd edzd e j dzdedzde jdzdeedefdZxZS) DbrxModelaTransformer decoder consisting of *config.num_hidden_layers*. Each layer is a [`DbrxBlock`] layer. Args: config ([`DbrxConfig`]): Model configuration class with all parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. r.c ,t|||j|_|j|_|j |_t ||_tj|j|j|j|_ tjt|jDcgc]}t||c}|_tj"|jd|_d|_|j)ycc}wr)r,r- pad_token_id padding_idx vocab_size emb_pdropr rotary_embr Embeddingr/wte ModuleListrangen_layersrblocksrnorm_fgradient_checkpointing post_initrs r%r-zDbrxModel.__init__fs  !.. ++))-f5<< 1 16>>4CSCSTmmSXY_YhYhSi$jiYvy%A$jk ll6>>> &+#  %ks4DrKc|jSrmr rCs r%get_input_embeddingszDbrxModel.get_input_embeddingsts xxr$valuec||_yrmrrCrs r%set_input_embeddingszDbrxModel.set_input_embeddingsws r$N input_idsrG position_idsrI inputs_embeds use_cacherJrDc D|du|duz r td|r|t|j}||j|}|F||j nd} t j | | |jdz|j}||jd}t|j|||||} |} |j| |} |jd|jjD]} | | f| | ||||d|} |j| } t| |S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)r)r. input_embedsrGrJrIr)rHrGrrIrrJ)last_hidden_staterI) ValueErrorr r.r get_seq_lengthrqarangerVr unsqueezer rrnum_hidden_layersrr)rCrrGrrIrrrJrDpast_seen_tokens causal_maskrFrH decoder_layers r%rlzDbrxModel.forwardzs^  -t"; <YZ Z 0*$++>O   HHY/M  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L(;;&))+%  & #oom\J![[)H4;;+H+HI M) $7*) /#-  M  M2 %++  r$)NNNNNNN)r r!r"rorr-rr rrrrrqrsrrr FloatTensorboolrrrrlrurvs@r%rr\s z bll",,.2.204(,26!%26; ##d*;  t+; &&- ;  ; ((4/ ; $;; ((4/; +,;  ; ; r$rc eZdZddiZddiZddgdgfiZdeffd Zd ejfd Z d ejfd Z d ejfd Z dejfdZdefdZd efdZee d dej*dzdej,dzdej*dzdedzdej0dzdej*dzdedzdedzdej*dzdeej,zdeed efdZxZS)!DbrxForCausalLMzlm_head.weightztransformer.wte.weightlm_headcolwise_gather_outputrFlogitsr.ct||t||_|j|_t j |j|jd|_|jj|_ |jj|_ |jj|_|j!yr)r,r-rrrrr@r0r-rmoe_loss_weightrouter_aux_loss_coefr}rrnum_experts_per_tokrrs r%r-zDbrxForCausalLM.__init__s  $V, ++yy!3!3V5F5FUS $*$5$5$E$E!!,,<<#)#4#4#>#>  r$rKc6|jjSrm)rrrs r%rz$DbrxForCausalLM.get_input_embeddingss4466r$rc:|jj|yrm)rrrs r%rz$DbrxForCausalLM.set_input_embeddingss --e4r$c|jSrmr-rs r%get_output_embeddingsz%DbrxForCausalLM.get_output_embeddingss ||r$new_embeddingsc||_yrmr7)rCr9s r%set_output_embeddingsz%DbrxForCausalLM.set_output_embeddingss % r$decoderc||_yrmr)rCr<s r% set_decoderzDbrxForCausalLM.set_decoders "r$c|jSrmr>rs r% get_decoderzDbrxForCausalLM.get_decodersr$NrrGrrIrlabelsroutput_router_logitsrJlogits_to_keeprDc l||n|jj}|jd|||||||| d| } | j} t | t r t | dn| }|j| dd|ddf}d}||j|||jfi| }d}|rYt| j|j|j|}|+||j|j|j zz }t#|||| j$| j&| j(| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, DbrxForCausalLM >> model = DbrxForCausalLM.from_pretrained("transformers-community/dbrx-instruct") >> tokenizer = AutoTokenizer.from_pretrained("transformers-community/dbrx-instruct") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ``` N)rrGrrIrrrCrJ)lossaux_lossr/rIrFrrr#)r.rCrr rrpslicer- loss_functionrrrrr3r2rrrrIrFr)rCrrGrrIrrBrrCrJrDrDoutputsrF slice_indicesr/rFrGs r%rlzDbrxForCausalLM.forwardsZP%9$D $++JjJj  +;$*:*: + )%+'!5) +  +  11 8B>SV8W~ot4]k mA}a,?@A  %4%%ffdooPPD /%%  (( H !11HKK 4LLL(#33!//))!//  r$) NNNNNNNNNr) r r!r"_tied_weights_keys_tp_plan_pp_planrr-rr rrr@r8r;rr?rArrrqrsrrr r)r*rprrrrlrurvs@r%r,r,s*,DE23H_-z:;Hz7bll75",,5ryy&BII&#9# Y .2.204(,26*.!%,026-.R ##d*R  t+R &&- R  R ((4/ R   4'R $;R #TkR ((4/R ell*R +,R  #R R r$r,)r,rr)8rocollections.abcrtypingrrqrrr activationsr cache_utilsr r generationr masking_utilsr modeling_layersr modeling_outputsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrllama.modeling_llamarrrmixtral.modeling_mixtralrconfiguration_dbrxrrrr'rxrrrrrrrr,__all__r#r$r%r`s)$ &!.)/RF&II/ @* . U)BIIU)pBII2$"))$N"bii6'.BII'.T!*!H7/74Z #Z Z zu )?u p Br$