i*dZddlZddlmZddlmZddlZddlmZddlm Z m Z m Z ddl m Zdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZmZmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+ddl,m-Z-e+rddl.m/Z/m0Z0ddl1m2Z2nd\Z2Z0Z/e*r ddl3m4Z4m5Z5nd\Z5Z4e6e2e0e4e5e/fZ7e&jpe9Z:GddejvZd ejzfd!Z?Gd"d#Z@ dCd$ejvd%ejzd&ejzd'ejzd(ejzdzd)eAd*eAfd+ZBGd,d-ejvZCGd.d/ejvZDGd0d1ejvZEGd2d3ejvZFGd4d5eZGGd6d7eZHe%Gd8d9e!ZIe%Gd:d;eIZJGd<d=eIeZKe%d>?Gd@dAeIZLgdBZMy)DzPyTorch Zamba model.N)Callable)Any)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging)is_flash_attention_requested)is_causal_conv1d_availableis_mamba_ssm_available) ZambaConfig)mamba_inner_fnselective_scan_fn)selective_state_update)NNN)causal_conv1d_fncausal_conv1d_updateNNc,eZdZdfd ZdZdZxZS) ZambaRMSNormct|tjt j ||_||_y)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s r/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/zamba/modeling_zamba.pyr(zZambaRMSNorm.__init__Bs1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor*float32powmeanrsqrtr-r,)r. hidden_states input_dtypevariances r2forwardzZambaRMSNorm.forwardJsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r3c^t|jjd|jS)Nz, eps=)tupler,shaper-r.s r2 extra_reprzZambaRMSNorm.extra_reprQs* ))*+6$2G2G1HIIr3)gư>)__name__ __module__ __qualname__r(rArF __classcell__r1s@r2r%r%As$;Jr3r%r>n_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rDexpandreshape)r>rLbatchnum_key_value_headsslenhead_dims r2 repeat_kvrUVso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr3c eZdZdZdZej dfdZdZ ddejdejde d e e e fdzd eejejff d Zd ej fd Zdde dzd e fdZy)ZambaHybridDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNc ,||_d|_|j|_d|_|j|j z|_|j|_|j|_ |j|_ g|_ g|_ g|_i|_i|_i|_t%|j&D]}|xjt)j*||j |j||gz c_ ||j|j |jz|jf}|xjt)j*|||gz c_ |j|dk(s|jj-|t%|j&Dcgc]}t)j.gg|z|c}|_t%|j&Dcgc]}t)j.gg|z|c}|_ycc}wcc}w)NFdevicer8hybridrZ)r8is_compileablelayers_block_typehas_previous_state mamba_expandr/intermediate_size mamba_d_statessm_state_size mamba_d_convconv_kernel_size n_mamba_heads conv_states ssm_statestransformer_layers_modules _parameters_buffersrangenum_hidden_layersr*zerosappendtensor key_cache value_cache)r.config batch_sizer8rZi cache_shape_s r2r(z ZambaHybridDynamicCache.__init__rs #!'!9!9"'!'!4!4v7I7I!I$22 & 3 3#11"$  v//0 2A    J(>(>@U@U^dlqr!  ""&&$*<*<<## K OO Ke TU UO%%a(H4''..q1 2SXX^XpXpRqrQ%,,tj'8HrTYZ`ZrZrTstqELL" ):6Jtsts "H !"Hc,t|jSN)lenrrrEs r2__len__zZambaHybridDynamicCache.__len__s4>>""r3 key_states value_states layer_idx cache_kwargsrMc|j|jddk(r||j|<||j|<nftj|j||gd|j|<tj|j||gd|j|<|j||j|fS)Nr6rr5dim)rrrDrsr*cat)r.r}r~rrs r2updatezZambaHybridDynamicCache.updates >>) $ * *2 .! 3(2DNN9 %*6D  Y '(- 4>>)3Lj2Y_`(aDNN9 %*/))T5E5Ei5PR^4_ef*gD  Y '~~i($*:*:9*EEEr3beam_idxc|jdkDrvtt|jD]S}|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<Vyy)zDReorders the cache for beam search, given the selected beam indices.rN) get_seq_lengthrmr{rrrZ index_selectr9rsrgrh)r.rrrZs r2 reorder_cachez%ZambaHybridDynamicCache.reorder_caches[    1 $"3t~~#67 m  299,0NN9,E,R,RSTV^VaVabhVi,jy))))4;;.2.>.>y.I.V.VWXZbZeZeflZm.n  +)))4;;.2.>.>y.I.V.VWXZbZeZeflZm.n  +3::-1__Y-G-T-TUVX`XcXcdjXk-l * m %r3c||jvr|jdn|}t|j|ks|j|jddk(ry|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.rr6)rir{rrrD)r.rs r2rz&ZambaHybridDynamicCache.get_seq_lengthsn3<4CZCZ2ZD++A.`i t~~ ) +t~~i/H/N/Nr/RVW/W~~i(..r22r3rz)r)rGrHrI__doc__r]r*float16r(r|TensorintdictstrrrCr LongTensorrrr3r2rWrWbs N16tu@#/3 FLLFllF F 38nt+ F u||U\\) * F$ me&6&6 m3d 333r3rWmodulequerykeyvalueattention_maskscalingdropoutc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr5r rr6)rr8)ptrainingr)rUnum_key_value_groupsr*matmul transposerDr functionalsoftmaxr:r9r8rr contiguous) rrrrrrrkwargsr}r~ attn_weights causal_mask attn_outputs r2eager_attention_forwardrs3 ; ;$>&B\B\$\!'-'E'E$ )d2 !'!9!9ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii : :T]] JFL^L^ejk r3Nr>rpast_key_valuesrrMc |jdd}g|d|j}|j|j|j dd}|j |j|j dd} |j |j|j dd} ||j| | |\} } tj|jjt} | ||| | |f|jsdn |j|jd|\} } | j g|dj#} |j%| } | | fS)Nr6rr5)rr)rDrTrviewrrrrr get_interfacert_attn_implementationrrrrrPrr)r.r>rrrr input_shape hidden_shape query_statesr}r~attention_interfacerrs r2rAzZambaAttention.forwardsk$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST  &'6'='=j,Xa'b $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r3rz)rGrHrIrrrr(r*rrWrrrCrArJrKs@r2rrs l{lsl.;? #)||#)#) t+ #) 147 #) -. #) u||U\\D0% 2E2LL M#)r3rcleZdZdZdeffd Z d dejdefdZ d defdZ d defdZ xZ S) ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. rtc t|||_||_|j|_|j |_|j|_|j|jz|_ |j|_ |j|_ |j|jz|_|j|_|j"|_t'j(|j|j|j |j|j|jdz |_|j,|_t0|j,|_|j4|_t'j8|j|jdz|j$|_t'j<t?j@|j|j|j dzz|j|_!t'j<t?j@|j|j|jdz dz|jdzz |_"t'j<t?j@|j|j|_#t?jHd|j dzt>jJdddf}|jM|jdjO}t'j<t?jP|jS|j|jd|_*t'j<t?jV|j|j|_,t'j8|j|j|j$|_-t\st^jadyy) Nr) in_channels out_channelsr kernel_sizegroupspaddingr5rg?r8r6aqThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)1r'r(rtrr/rbrcrdrer`ra mamba_dt_ranktime_step_rankrfmamba_head_dimmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1dhidden_mamba_act activationr actuse_mamba_kernelsuse_fast_kernelsrin_projr)r*ro x_proj_weightdt_proj_weight dt_proj_biasaranger:rOrlogrPA_logr+Dout_projis_fast_path_availablelogger warning_once)r.rtrAr1s r2r(zZambaMambaMixer.__init__-s   "!--$22 & 3 3!'!4!4v7I7I!I$22#11"448J8JJ#33.. ii..//##--))))A-   !11&112 & 8 8yy!1!143I3IA3MTXTaTab  \\ KK""##d&9&9A&==##  !ll [[++T-@-@$BUBU VY\ \ !!3& '  LLT5G5GI\I\)]^ LLD//!35== I$PQ' R HHT++R 0 ; ; =\\%))A,"6"6t7I7I4K^K^`b"cd ejj););T=P=PQR $"8"8$:J:JQUQ^Q^_ %   ^ &r3r> cache_paramsc |j\}}}|duxr|jxr|dk(}|j|jdd}|j |dd|j dd\}} |j dj}| j d} | j||jd|jdd} |jjj |jjjd|jjjd} |ret|j d|j|j| |jj |j"}|j%d}n|,t'j(|dk(s||j%dz}|dt*j,j/||j0|jdz df} |j|jj3| t5|| |jj |j"}|,t'j(|dk(s||j%dz}|jd|j|j6|jdd}|j8dddddddf|zjdd} t'j:| |j<|j>|j>gd\} }}|j@dddf| jddz}t'jB|jDjG }|jH|jHjGnd}t'jJ|d|f|jL|jN}|rtQ|jD]}tS|jT|jdd|f||d df||d df||||dddf||dddf|jV|| |d df||d j%d}t'jX||fd}nAt'jJ|d|j6|j>f|jL|jN}tQ|jD]}t[||||||||jdd||jdd|jV|jG| |||d d \}}t'jX||fdj}t'jX||j%dfd}|*|(|jT|jj3||j]|jdd}|S) Nrr5r6rr)rrrY.T) dt_softplus)delta_softplusreturn_last_state)/rDr_rrrchunksqueezerrPrfrr,sizer"rgrrr unsqueezer*allrrpadrecopy_r!rrsplitrrcrexprfloatremptyrZr8rmr rhrrrr)r.r>rrruseq_lenrxuse_precomputed_statesprojected_statesgate conv_weightsrgssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statess r2cuda_kernels_forwardz$ZambaMambaMixer.cuda_kernels_forwardjs"/!4!4 GQ!-T!9!nl>]>]!nbimnbn << 6@@AF.33JAwOUUVW]^U_ t%--a0;;= ||A||J(:(:BHRRSTVWX{{))..t{{/A/A/F/Fq/I4;;K]K]KbKbcdKef !0%%b)((8    M*33B7M)%))Na>{K,]L$++JZJZgkgvgvwM)%))Nar use_cacher conv_staterrrrrr discrete_A discrete_BdeltaB_urrv scan_outputr s r2 slow_forwardzZambaMambaMixer.slow_forwards!-!3!3 GQ""<< 5??1E.33JAwOUUVW]^U_ t%--a0;;= ||A||J(:(:BHRRSTVWX|-DE 00@FFqIZW}}(33DNNCIIK (33DNNC ! ]%9%9:I//qL ,,T^^<BB1ES)55dnnE "ZZ 2BG '4Q1W'= 1a8$;E ((8 % *t{{7I7I!QPQ'7R*RXZ [ %%!T[[%5%55M $ 7 : :5 A K KB O !-$1N1}GZGZ[]G^F^F`C`4a4k4klm4n$nM]]..}t?T?TWdWjWjkmWn?npq>rs ;E ((8 $])CC'M)R S !-$1N1}GZGZ[]G^F^F`C`4a4k4klm4n$nM T//1D1DdFYFYZ$++I ) -0H0H0K K  HHT[[%?XgX %NOM) -0H0H0K K &--b$2D2DdFYFY[bcmmnoqrs ,,Qa];mKVVWY[]^++ T00$2E2EtGZGZ[ac  1a#11!T':Y=P=PQSUW=XX\`\m\m tQ ]   ]]334FGYYtzz'') * *YYqD!T1!458J1aQRTUW[K[8\\] '1aD(89AaD!Q>N ? E E GG w 9A"1aAq=1;;AqAIMPXYZ\]_`bcefYfPgPqPqrsuvPwwI,,y':':1a'@'C'CE'JAaQRTUWXjMLcLcdfLghK    Aq!QJ 7 8 9kk,B7 !]TVVAtQ=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r)rrrrZtype ValueErrorr r)r.r>rrs r2rAzZambaMambaMixer.forward(sm  )V4;M;M;T;T;Y;Y-Y i ,,]LYg,h h   ^ \\r3r#) rGrHrIrrr(r*rrWr rrArJrKs@r2rr sX ;{;|im_%"\\_%9P_%B[%7N[%z ]3J ]r3rc$eZdZfdZdZxZS)ZambaMLPct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFr) r'r(rtr/rarr gate_projup_proj down_projr hidden_actact_fnr.rtr1s r2r(zZambaMLP.__init__6s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r3c|j|j|j||j|z}|Srz)r'r)r%r&)r.xr's r2rAzZambaMLP.forward@s6NN4;;t~~a/@#ADLLQRO#ST r3)rGrHrIr(rArJrKs@r2r"r"5s 0r3r"ceZdZddededzffd Z ddejdejdedejdzdedzd e dzd e dzd e e d e eje ejejfdzffd ZxZS)ZambaAttentionDecoderLayerNrtrct|t|||_t ||_t |j|j|_ t |j|j|_ y)Nr0) r'r(r self_attnr" feed_forwardr%r rms_norm_epsinput_layernormr/pre_ff_layernormrs r2r(z#ZambaAttentionDecoderLayer.__init__Fs_ ' :$V,+F,H,HfNaNab ,V-?-?VEXEX Yr3r>original_hidden_statesrroutput_attentionsrrrMc tj||gd}|j|}|jd||||||d|\}} |j |}|j |}|f} |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. r6r)r>rrrr7rr)r* concatenater4r1r5r2) r.r>r6rrrr7rrself_attn_weightsoutputss r2rAz"ZambaAttentionDecoderLayer.forwardNs>))=:P*QWYZ ,,]; +94>>, ')+/ , , ( (--m< ))-8  "  )+ +Gr3rz)NNFF)rGrHrIrrr(r*rrWboolrrrC FloatTensorrArJrKs@r2r.r.EsZ{ZsTzZ/3:>).!&3||3!& 3 3  t+ 3 147 3 $;3$;3-.3 u  %(9(95;L;L(L"MPT"TT U3r3r.ceZdZdedeffd Z ddejdejdzdedzdejdzdejdzd edzd e dzd e dzd ejdzd ejdzdejdzde eje ejejfdzffdZ xZS)ZambaMambaDecoderLayerrtrct|t|||_t |j |j |_||_y)N)rtrr0) r'r(rmambar%r/r3r4rrs r2r(zZambaMambaDecoderLayer.__init__s> $FiH +F,>,>FDWDWX"r3Nr>r6rrrr7rcache_position position_idstransformer_hidden_statesrMc  |} | || zn|}|j|}|j|||}d}| |z}|f}|r||fz }|r||fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. N)r>rr)r4rA)r.r>r6rrrrr7rrBrCrDrresidualr:r;s r2rAzZambaMambaDecoderLayer.forwards>! :S9^M5 5dq ,,];  '()# !!=0  "  )+ +G  ) )Gr3) NNNNNFFNNN)rGrHrIrrr(r*rrWr<rrCr=rArJrKs@r2r?r?s,#{#s#7; $.2+/:>).!&26049=;||;!& t 3;: ;  t+ ; \\D( ;147; $;;$;;((4/;&&-;$)<<$#6; u  %(9(95;L;L(L"MPT"TT U;r3r?cleZdZdedej deffd Z ddejdejdzde dzd ejdzd ejdzd e dzd e dzd e dzdejdzdeejeejejfdzffdZxZS)ZambaHybridLayer shared_transflinearrAcLt|||_||_||_yrz)r'r(rIrJ mamba_decoder)r.rIrJrAr1s r2r(zZambaHybridLayer.__init__s% * "r3Nr>r6rrrrr7rrBrMc |j|||||||| } | d} |r| d} |j| } |j|| ||||| } |r | d f| ddz} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )r6rrrr7rrBrr)rDrrr7rrBr5N)rIrJrL) r.r>r6rrrrr7rrB layer_outputsrDr:s r2rAzZambaHybridLayer.forwards>** #9&+/)+  %2!$4!  -a 0 $(KK0I$J!** &?)+/)+  *1-/@AMRSRTDUUMr3)NNNNNFFN)rGrHrIr.rrr?r(r*rrrWr<rrCr=rArJrKs@r2rHrHs #&@#"))#\r#7; $.2+/:>).!&26>||>!& t 3>: >  t+ > \\D( >147> $;>$;>((4/> u  %(9(95;L;L(L"MPT"TT U>r3rHcreZdZUeed<dZdZddgZdZdZ dZ dZ e jfdZxZS) ZambaPreTrainedModelrtmodelTr.r?rFc|jj}t| |t |t rt j|jd||jjdz}t j|j| ||jj|jjz|jjz}tj tj"|jj|t%j&|jj(t%j&|jj*z zt%j&|jj*zj-|jj.}|tj&tj0|  z}t j2|j4|tj6d|j8dztj:dddf}|j=|j>djA}t j2|jBtj&|jE|j|jFdt jH|jJyy)Nr)r<stdr)minrrr6)&rtinitializer_ranger' _init_weightsrrinitnormal_rruniform_rr`r/rfr*rrandmathr time_step_max time_step_minclamptime_step_floorexpm1rrrrcr:rOrarrrPrones_r) r.rrS dt_init_stdrdtinv_dtrr1s r2rVz"ZambaPreTrainedModel._init_weightsskk++ f% fo . LL--CS A++33T9K MM&//+{ K![[55 8O8OOSWS^S^SlSllN 4;;44nE88DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F JJv**F 3 Q 5 5 9OPTVWPWXA1126AACA JJv||UYYq\%9%9&:N:NPVPePegi%j k JJvxx % /r3)rGrHrIr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulr*no_gradrVrJrKs@r2rPrPsQ &*#57OP"3 NLU]]_!!r3rPc(eZdZdZdeffd Ze ddejdzdejdzdejdzde dzd ejdzd e dzd e dzd e dzd e dzdejdzde ezfdZdZxZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rtcht|||j|_|j|_t j |j|j|j|_|j|_ g}d|_ t|jD]\}}t||}|dk(rt j|jj|jjd}|jt!t#||||jd|dd|di|_ |j|t j$||_|j(|_t+|j|j, |_d|_|j3y) N)rr[Frz layers.(?!z\.)\d+.shared_transfzlayers.z.shared_transfr0)r'r( pad_token_id padding_idx vocab_sizer Embeddingr/ embed_tokensr^_tied_weights_keys enumerater?rrtrprHr. ModuleListlayersrr%r3final_layernormgradient_checkpointing post_init)r.rtrylayer_id layer_typerArJr1s r2r(zZambaModel.__init__?sq  !.. ++LL):):F,>FDWDWX&+# r3N input_idsrrCr inputs_embedsrr7output_hidden_states return_dictrBrMc ||n|jj}||n|jj}||n|jj}| | n|jj} |du|duz r t d|j r%|jr|rtjdd}||j|}|} tj|} |r|tjd| .tj| jd| j} || j!d}|j#||| }|rdnd}|rdnd}t%|j&D]5\}}|r|| fz }|| | ||||||| }|d} |s'|d-||dfz }7|j)| } |r|| fz }|r|j*sd |_t-| |r|nd|| }| r|S|j/S) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fz{Zamba requires an initialized `ZambaHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.rr\rr)rr7rrBT)last_hidden_staterr> attentions)rtr7rruse_return_dictr r{rrrrur*rrrDrZr_update_causal_maskrwryrzr_rto_tuple)r.rrrCrrrr7rrrBrr>r6rall_hidden_statesall_self_attnsrlayerrNoutputs r2rAzZambaModel.forward\sI2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK26!%)-,0#'26_<##d*_< t+_<&&- _< 147 _< ((4/ _<$;_< $;_<#Tk_<D[_<((4/_< ( (_<_>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrCrrrr7rrBrrrlosslogitsrr>r)rtr7rrrQrrslicer loss_functionrsrrr>r)r.rrrCrrrrr7rrrBrrr;r> slice_indicesrrrs r2rAzZambaForCausalLM.forwardsLP2C1N-TXT_T_TqTq%9$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %4%%ffdooPPDY,F'+'7D7V# CV C%#33!//))   r3c |:t|j|jd|j|j}|jj | d<t ||f|||||||d| } | S)Nr)r8rZr)rrrrBrCris_first_iteration)rWrtrDr8rZnum_logits_to_keepr'prepare_inputs_for_generation) r.rrrrrBrCrrr model_inputsr1s r2rz.ZambaForCausalLM.prepare_inputs_for_generationEs  "5 Y__Q/tzz$++O$(;;#A#A w<   +)')%1    r3) NNNNNNNNNNNr)NNNNNTF)rGrHrIrvrr(rr*rrrWr=r<rrCrrArrJrKs@r2rrsj*,GH{.2.204:>26*.!%)-,0#'26-.O ##d*O  t+O &&- O 147 O ((4/ O   4'O $;O  $;O #TkO D[O ((4/O ell*O  ' 'O O h   r3ra The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introceZdZfdZe ddej dzdejdzdej dzdedzdejdzdej dzd e dzd e dzd e dzd e dzd e e zfdZ xZS)ZambaForSequenceClassificationct|||j|_t||_t j |j|jd|_|jyr$) r'r( num_labelsrorQrrr/scorer|r*s r2r(z'ZambaForSequenceClassification.__init__wsS   ++' YYv114??O  r3NrrrCrrrrr7rrrMc | | n|jj} |j|||||||| |  } | d} |j| }||jd}n|jd}|jj |dk7r t d|jj d}n|||jj k7j|jtj}tj|jd|jtj}||zjd}n.d}tj|jj d|tj||j |f}d}||j|j}|jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}| s|f| ddz}||f|zS|St7||| j8| j:| j< S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrCrrrr7rrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r6rYz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r\ regressionsingle_label_classificationmulti_label_classificationr)rtrrQrrDrqr r9rZr*int32rargmaxrrr1rG problem_typerr8longrrrrrrrrr>r)r.rrrCrrrrr7rrrtransformer_outputsr>rrulast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r2rAz&ZambaForSequenceClassification.forwards*&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r3r)rGrHrIr(rr*rrr r=r<rCrrArJrKs@r2rrhs.2.204(,26*.!%)-,0#'\ ##d*\  t+\ &&- \  \ ((4/ \   4'\ $;\  $;\ #Tk\ D[\  1 1\ \ r3r)rrrorP)r)Nrr[collections.abcrtypingrr*rtorch.nnrrrr rW activationsr cache_utilsr generationr modeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrr utils.genericrutils.import_utilsrrconfiguration_zambar&mamba_ssm.ops.selective_scan_interfacerr+mamba_ssm.ops.triton.selective_state_updater causal_conv1dr!r"rr get_loggerrGrModuler%rrrUrWrrrrr"r.r?rHrProrr__all__rr3r2rsD& $ AA&! )>B9qqF&,9T,XR@P=-~DD-7**.0@BVXfg   H %J299J* UU\\ U# U%,, U[3[3J% II% <<% % << % LL4' %  %%4C)RYYC)LQ]biiQ]j ryy  <<~B7BJE1EP!!?!!!!Hl%ll`~+_~B  g %9g  g T gr3