i2ddlZddlmZddlmZddlmZmZddlZddl m Z ddl m cm Z ddl mZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6ddl7m8Z8m9Z9m:Z:edGdde jvZ<Gdde jvZ=Gdde jvZ>Gdde jvZ?Gd d!e jvZ@Gd"d#e jvZAd$ZBd%ejd&ejd'ejd(ejd)eDejejff d*ZEd+ejd,eFd)ejfd-ZG dWd.e jvd/ejd0ejd1ejd2ejdzd3eHd4eHd5e,e.fd6ZIGd7d8e jvZJGd9d:e ZKGd;dZNGd?d@e jvZOGdAdBe jvZPGdCdDe ZQee/dEFGdGdHe$ZRe/GdIdJe*ZSGdKdLeSZTe/GdMdNeSZUe/GdOdPeSZVee/dQFGdRdSe$ZWGdTdUeSeZXgdVZYy)YN)Callable) dataclass)AnyOptional) LayerNorm)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPooling ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilingtorch_compilable_check)check_model_inputsis_flash_attention_requestedmaybe_autocast) Glm4vConfigGlm4vTextConfigGlm4vVisionConfigRMSNormc,eZdZdfd ZdZdZxZS) Glm4vRMSNormct|tjt j ||_||_y)z; Glm4vRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s r/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/glm4v/modeling_glm4v.pyr+zGlm4vRMSNorm.__init__7s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NTkeepdim) dtypetor.float32powmeanrsqrtr1r0)r2 hidden_states input_dtypevariances r6forwardzGlm4vRMSNorm.forward?sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r7c^t|jjd|jS)Nz, eps=)tupler0shaper1r2s r6 extra_reprzGlm4vRMSNorm.extra_reprFs* ))*+6$2G2G1HIIr7)gư>)__name__ __module__ __qualname__r+rFrK __classcell__r5s@r6r(r(5s$;Jr7r(c,eZdZddeffd ZdZxZS) Glm4VisionMlpbiasct||j|_|j|_t j |j|j||_t j |j|j||_t j |j|j||_ t|j|_ yNrS) r*r+r3out_hidden_sizeintermediate_sizer,Linear gate_projup_proj down_projr hidden_actact_fn)r2configrSr5s r6r+zGlm4VisionMlp.__init__Ks !--!'!7!74#3#3T5K5KRVWyy!1!143I3IPTU 4#9#94;K;KRVWV../ r7c|j|j|j||j|zSN)r\r^rZr[r2 hidden_states r6rFzGlm4VisionMlp.forwardTs2~~dkk$..*FG$,,WcJddeer7F)rLrMrNboolr+rFrOrPs@r6rRrRJs0T0fr7rRc`eZdZdeddffd Zdej dej fdZxZS)Glm4vVisionPatchEmbedr_returnNcTt||j|_|j|_|j|_|j |_|j|j|jg}tj|j|j |||_ y)N) kernel_sizestride) r*r+ patch_sizetemporal_patch_size in_channelsr3 embed_dimr,Conv3dproj)r2r_rjr5s r6r+zGlm4vVisionPatchEmbed.__init__Ys  ++#)#=#= !--++//$//R IId..K`kl r7rCc6|jjj}|jd|j|j |j |j }|j|j|jd|j}|S)Nr:r=) rqr0r=viewrnrmrlr>ro)r2rC target_dtypes r6rFzGlm4vVisionPatchEmbed.forwardcs~yy''-- %**   $":":DOOT__  -"2"2"2"FGLLRQUQ_Q_` r7 rLrMrNr%r+r.TensorrFrOrPs@r6rgrgXs5m0mTmU\\ellr7rgcreZdZUejed<d dededdffd ZdedejfdZ xZ S) Glm4vVisionRotaryEmbeddinginv_freqdimthetarhNct|||_||_d|t j d|dtj |z zz }|jd|dy)N?rr9rsrzF persistent)r*r+r{r|r.arangefloatregister_buffer)r2r{r|rzr5s r6r+z#Glm4vVisionRotaryEmbedding.__init__osY  %ELLC%++$NQT$TUV ZeDr7seqlenctj||jj|jj}tj ||j}|S)Ndevicer=)r.rrzrr=outer)r2rseqfreqss r6rFz"Glm4vVisionRotaryEmbedding.forwardvsAll6$--*>*>dmmFYFYZ C/ r7)g@) rLrMrNr.rw__annotations__intrr+rFrOrPs@r6ryrylsBllECEEDEcellr7ryc neZdZd dededededdf fd Zdejdejfd Z xZ S) Glm4vVisionPatchMergerr{ context_dimr]rSrhNcxt|tj||||_t ||_tj||||_tj||||_tj||||_ tj|_ t||_ yrU)r*r+r,rYrqrpost_projection_normrZr[r\GELUact1r r^)r2r{rr]rSr5s r6r+zGlm4vVisionPatchMerger.__init__}s IIc3T2 $-cN!3 $?yyk= ;$?GGI Z( r7rcc|j|}|j|j|}|j|j |j ||j |zSra)rqrrr\r^rZr[rbs r6rFzGlm4vVisionPatchMerger.forwardsYyy. yy!:!:1D!--"$,,t/A/A4>>"R#, r7rhc 2|jj}|jd}|j}t |t r&t j||t j}|jd} t| dz} |j| | |jdddjdj|t j} t jt!t#|D cgc]} || dfj%|| c} j|t j} t jt!t#|D cgc]} || dfj%|| c} j|t j}|dz|z dzdz }|dz| z dzdz }t j&||fdjdjd}t)j*| ||j,dd }|j/dj/djdd}|j|j0j|j}||z}|Scc} wcc} w) a Forward pass with integrated position encoding adaptation using 2D interpolation. Args: embeddings: Input embeddings tensor lengths (torch.Tensor): Sequence lengths for each image in the batch. image_shapes (torch.Tensor): Tensor of shape [batch_size, 3] representing the image shapes (t, h, w). h_coords (torch.Tensor): Tensor of shape [total_seq] representing the h coordinate for each patch. w_coords (torch.Tensor): Tensor of shape [total_seq] representing the w coordinate for each patch. Returns: torch.Tensor: Embeddings with adapted position encoding added. r"rrg?r9r:r{Fborder)mode align_corners padding_mode)rr0rIr isinstancelistr.tensorlongrrtpermute unsqueezer>r?catrangelenrepeatstackF grid_samplersqueezer=)r2 embeddingslengths image_shapesh_coordsw_coordspos_embed_weightr3r orig_size_sq orig_size pos_embed_2ditarget_htarget_wnorm_wnorm_hgridinterpolated_embed_fp32adapted_pos_embed_fp32adapted_pos_embeds r6rFzGlm4vVisionEmbeddings.forwards^ 2299&,,Q/ !(( gt $ll76LG(--a0  c)*  ! !)Y D WQ1  Yq\ RvU]]R 3 99USVW^S_M`al1a4077 Cabeef 99USVW^S_M`al1a4077 Cabeef  c>X-2Q6c>X-2Q6{{FF+4>>qAKKAN#$-- $T%=%=Uai#  "9!@!@!C!K!KB!O!W!WXY[\!]2556F6L6LMPPQ[QbQbc "33 3bbs < J' JrvrPs@r6rrs# -0 -;PUP\P\;r7rc|dd|jddzf}|d|jddzdf}tj| |fdS)*Rotates half the hidden dims of the input..Nr:r9r)rIr.rxx1x2s r6 rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r7qkcossinrhc|j}|j}|j|j}}|jdj|jdj}}||zt||zz}||zt||zz}|j |}|j |}||fS)N)r=rrrr>)rrrr orig_q_dtype orig_k_dtypeq_embedk_embeds r6apply_rotary_pos_emb_visionrs77L77L 779aggiqA}}R &&(#--*;*A*A*CC3w;q>C/0G3w;q>C/0Gjj&Gjj&G G r7rCn_repc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r"N)rIexpandreshape)rCrbatchnum_key_value_headsslenhead_dims r6 repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr7modulequerykeyvalueattention_maskscalingdropoutkwargsc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr9rrr:r{r=)ptrainingr")rnum_key_value_groupsr.matmul transposerIr, functionalsoftmaxr?r>r=rr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r6eager_attention_forwardrs3 ; ;}}}}t'j.|d }|j|dj1}|j3|}|Scc}wcc}}}w) Nrrr:r"r9F)rrr cu_seq_lens_q cu_seq_lens_k max_length_q max_length_krr)rrrr)rIrrrrunbindrrrr get_interfacer__attn_implementationrr maxrrrr.splittolistziprrrq)r2rCrrrr seq_length query_statesrrrrattention_interface max_seqlenr_rrsplitsrrv attn_outputss r6rFzGlm4vVisionAttention.forward!s#((+ HH] # + +J4>>2 N V VWXZ[]^`a b i ijk l / j,'S#>|ZY\^a#b j#--a3==a@ ))!Q/99!< #--a3==a@ (?(M(M KK , ,.E)  ( 4$QR.:cr?:??AJ0  $ #'==d6L6L((''NK"!nz#26GLXZdfrKsAG FGNN$4!<F #F|  Aq!$  $( LL'+}}C$:P:P#    L  ))La8K!))*b9DDF ii , - s -I?AINN) rLrMrNr%r+r.rwrHrFrOrPs@r6rrs 0 T "/3HL B||BLLB t+ B #5<<#=>E B Br7rc eZdZd fd Z d dej dej dej dzdeej ej fdzdej f dZxZS) Glm4vVisionBlockrhNct|t|j|j|_t|j|j|_t||_t|d|_ y)Nr4FrV) r*r+r(r3 rms_norm_epsnorm1norm2rattnrRmlprs r6r+zGlm4vVisionBlock.__init__gs\ !&"4"4&:M:MN !&"4"4&:M:MN (0  e4r7rCrrrc ||j|j|f|||d|z}||j|j|z}|S)N)rrr)r#r!r$r")r2rCrrrrs r6rFzGlm4vVisionBlock.forwardnsc&  JJ} %) !) 3 )  )  &M1J(KK r7rhNr) rLrMrNr+r.rwrHrFrOrPs@r6rrfsq5/3HL ||LL t+  #5<<#=>E  r7rc eZdZUejed<d deffd Ze ddedzde dde dzde d e ffd Z ejed Zd ZxZS)Glm4vTextRotaryEmbeddingrzNr_ct||j|_|j|_||_|j j d|_|j}|jdk7rt|j}||j |\}|_ |jd|d|jd|jd|j jdgd|_y) N rope_typedefaultrzFroriginal_inv_freq mrope_section) r/)r*r+max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr_rope_parametersr*compute_default_rope_parametersrattention_scalingrclonegetr-)r2r_r rope_init_fnrzr5s r6r+z!Glm4vTextRotaryEmbedding.__init__s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuU#3377Ur7rz torch.deviceseq_lenrhz torch.Tensorcn|jd}|jjdd}t|ddxs|j|jz}t ||z}d}d|t jd|dt jj|t j |z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorr~rNrr9rsr) r3r7getattrr3num_attention_headsrr.rint64r>r) r_rr9baser<rr{attention_factorrzs r6r4z8Glm4vTextRotaryEmbedding.compute_default_rope_parameterss&%%l3 & 6 6 : :;RTW X6:t4h8J8JfNhNh8h(223 U\\!S!5;;?BB&X]XcXcBdgjj k )))r7c^|jdddddfjjd|jddd}|dddddddfj}t |j j tr/|j j dk7r|j j nd}t|d5|j|jzjdd}|j||j}tj||fd }|j|jz}|j!|jz} dddj#|j$  j#|j$ fS#1swYAxYw) Nrr"r:mpscpuF) device_typeenabledr9rrs)rzrrrIrrtyperr!r apply_mroper-r.rrr5rr>r=) r2r position_idsinv_freq_expandedposition_ids_expandedrErembrrs r6rFz Glm4vTextRotaryEmbedding.forwards` !MM$a*=>DDFMMaQ]QcQcdeQfhjlmn ,Q4] ; A A C'1!((--'E!((--[`J`ahhmmfk  U C 5&,,.1F1L1L1NNYYZ[]^_E$$UD,>,>?E))UEN3C'')d444C'')d444C  5vvAGGv$cff177f&;;; 5 5s B!F##F,c |}|j|d}tjt|Dcgc] \}}||dzc}}d}|Scc}}w)Nr:rr)rr.r enumerate)r2rr-sectionchunksrchunkresults r6rHz$Glm4vTextRotaryEmbedding.apply_mropesQW"-69JKXQE!a%LKQST LsA ra)NNN)rLrMrNr.rwrr$r+ staticmethodrrrHrr4no_gradrrFrHrOrPs@r6r(r(sllVV")-+/"*$&*(*t* ~u$ % **>U]]_<< r7r(c||ddddf}|ddddf}tj| |fdjdS) r.rNr9r"r:rr)r.rflattenrs r6rotate_half_llmrWsJ 319B 319B ;;Ryb ) 1 1" 55r7c|j|}|j|}|dd|jddzfjdd}|dd|jddzfjdd}|jd}|dd|f|d|df}}|dd|f|d|df} }||zt||zz} ||zt||zz} t j | |gd} t j | | gd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nr:r9r)rrIrepeat_interleaverWr.r) rrrr unsqueeze_dim rotary_dimq_rotq_passk_rotk_passrrs r6apply_rotary_pos_embr`sD$ -- &C -- &C c'SYYr]a''' ( : :1" : EC c'SYYr]a''' ( : :1" : EC2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{u5;rrrrrrr3rr,rYq_projk_projv_projo_projr2r_rcr5s r6r+zGlm4vTextAttention.__init__sI  "!--33((DNN: #)#=#= $(NNd6N6N$N!!'!9!9%55}}d* ii 0 0$..4==2PW[\ ii 0 0$2J2JT]]2Zaef ii 0 0$2J2JT]]2Zaef ii >@P@PW\] r7rCrrpast_key_valuescache_positionrrhc T|j\}}} |j|} |j|} |j|} | j ||d|j j dd} | j ||d|j j dd} | j ||d|j j dd} |\} }t| | | |\} } |'|| |d}|j| | |j|\} } tj|jjt}||| | | |f|jsdn |j |j"d|\}}|j%||dj'}|j)|}||fS)Nr:r"r9)rrrkr)rr)sizererfrgrtrrr`updatercrr r_rrrrrrrrh)r2rCrrrjrkrbszq_lenrrrrrr cache_kwargsrrrs r6rFzGlm4vTextAttention.forwards&**, UA{{=1 [[/ {{=1 #((eRGQQRSUVW __S%T]]CMMaQRS #((eRGQQRSUVW &S#7 jRUWZ#[ j  &#&snUL'6'='=j,X\XfXfht'u $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \"))#ub9DDF kk+. L((r7raNNNN)rLrMrN__doc__r$rr+r.rwrHr LongTensorrrrFrOrPs@r6rbrbs ^^3:^.IM.2(,26 +)||+)#5<<#=>E+) t+ +)  +) ((4/ +)-.+) u||U\\D0% 2E2LL M+)r7rbcVeZdZfdZdej dej fdZxZS) Glm4vTextMLPc*t|||_tj|j d|j zd|_tj|j |j d|_t|j|_ y)Nr9FrV) r*r+r_r,rYr3rX gate_up_projr\r r] activation_fnrs r6r+zGlm4vTextMLP.__init__Fsp  IIf&8&8!f>V>V:V]bc6#;#;V=O=OV[\#F$5$56r7rCrhc|j|}|jdd\}}||j|z}|j|S)Nr9r:r)rxrQryr\)r2rC up_statesgates r6rFzGlm4vTextMLP.forwardNsL%%m4 #//!/4i 2 24 88 ~~i((r7)rLrMrNr+r. FloatTensorrFrOrPs@r6rvrvEs'7)U%6%6)5;L;L)r7rvcdeZdZdedeffd Ze ddejde ejejfdzdejdzdejdzd e dzd e dzd ejdzd e eje ejejfdzffd ZxZS)Glm4vTextDecoderLayerr_rcct||j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ y)Nr) r*r+r3rb self_attnrvr$r(r input_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernormris r6r+zGlm4vTextDecoderLayer.__init__Xs !--+FI>'+F,>,>FDWDWX(4V5G5GVM`M`(a%(4V5G5GVM`M`(a%".v/A/AvGZGZ"[r7NrCrrrIrj use_cacherkrhc |} |j|}|jd|||||||d|\}} |j|}| |z}|} |j|}|j |}|j |}| |z}|S)N)rCrrrIrjrrk)rrrrr$r) r2rCrrrIrjrrkrresidualrs r6rFzGlm4vTextDecoderLayer.forwardbs!,,]; *4>>  ' 3)%+)     q55mD  =0 !55mD / // >  =0 r7)NNNNFN)rLrMrNr$rr+rr.rwrHrtr rer}rFrOrPs@r6rrWs\\3\IM.204(,!&26%||%#5<<#=>E% t+ % &&- %  %$;%((4/% u  %(9(95;L;L(L"MPT"TT U%%r7rzJ Base class for Llava outputs, with hidden states and attentions. ) custom_introceZdZUdZdZej dzed<dZe dzed<dZ e ej dzed<dZ e ej dzed<dZ ejdzed<y)Glm4vModelOutputWithPasta[ past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlast_hidden_staterjrC attentions rope_deltas)rLrMrNrsrr.r}rrjr rCrHrrrtrr7r6rrsv37u((4/6$(OUT\(59M5**+d2926Je''(4/6+/K!!D(/r7rc\eZdZUeed<dZdZdZddgZdZ dZ dZ dZ dZ eedZfd ZxZS) Glm4vPreTrainedModelr_model)imagevideotextTrrrjrCrc"t||t|trod|jt j d|jdt j|jz zz }tj|j|yy)Nr~rr9rs) r* _init_weightsrryr|r.rr{rinitcopy_rz)r2rrzr5s r6rz"Glm4vPreTrainedModel._init_weightssk f% f8 9fllu||Avzz1TYT_T_/`cicmcm/mnoH JJv 1 :r7)rLrMrNr#rbase_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendrrb_can_record_outputsrrOrPs@r6rrs\ 1&*#02DE"3N!"&.( 22r7rc eZdZUeed<dZdgZeedZ d fd Z dZ e e dejd ejd eedeezfd ZxZS) Glm4vVisionModelr_)rrrrrhcFt|||j|_|j|_t ||_t ||_|j|jz}t|dz|_ tjt|jDcgc] }t!|c}|_t%|j&|j(|j*|_t/|j|j0|_tj4|j|j&|j|j|_t/|j|j0|_d|_|j=ycc}w)Nr9)r{rr]r)rn out_channelsrjrkF)r*r+spatial_merge_sizerlrrrg patch_embedr3rryrr, ModuleListrdepthrblocksrrWrXr]mergerr(r post_conv_layernormConv2d downsamplepost_layernormgradient_checkpointing post_init)r2r_rrr5s r6r+zGlm4vVisionModel.__init__sA  "(";"; ++/708%%)9)998QGmmuV\\GZ$[!%5f%=$[\ ,&&F4L4LY_YjYj  $00B0BH[H[#\ ))**//11,,   +6+=+=6CVCVW&+# %\s%Fcg}|D]s\}}}tj|jdjd|}|j ||j z|j ||j z|j }|j dddd}|j}tj|jdj|d}|j ||j z|j ||j z|j }|j dddd}|j}|jtj||gdj|dvtj|d}|ddddfj}|j|} | |jd} | |fS)Nr"r:rr9rr)r.rrrrrrrVappendrrrrr) r2grid_thwpos_idsthwhpos_idswpos_ids max_grid_sizerotary_pos_emb_fullrs r6 rot_pos_embzGlm4vVisionModel.rot_pos_embs SGAq!||A003::2qAH''T,,,''T,,,'' H  ''1a3H'')H||A003::1bAH''T,,,''T,,,'' H  ''1a3H'')H NN5;;(';DKKAqQ R) S*))G+ AB++- "11-@,W5==a@w&&r7rCrrc n|j|}|j|}|j|\}}tj||fd}|j |j f}tj|dddf|dddfz|dddfjdtjjr |jntj}tj|dd }|dd|ddz j} |j!|| ||dddfj#|j$|dddfj#|j$}|j&D]} | |f||d |}|j)|}|j+d|j,|j,|j.d}|j1dd dd}|j3|j+d|j4j6}|j9|} t;|| S) a\ hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`): The final hidden states of the model. grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`): The temporal, height and width of feature shape of each image in LLM. Returns: `torch.Tensor`: hidden_states. r:rNr"r9rr)r"r)r)rrr)r pooler_output)rrrr.rrrrYcumsumjit is_tracingr=int32rpadrrr>rrrrtrrIrrr_rWrr) r2rCrrrimage_type_idsrLrrseqlensblkmerged_hidden_statess r6rFzGlm4vVisionModel.forwards&((7 00? )-)9)9()C&ii8bA"wwy#'')4,,Xad^hq!tn-LhWXZ[W[n]dd %*II$8$8$:(.. e UU:vQ7 ab>JsO3;;=    1a4 # #M$8$8 9 1a4 # #M$8$8 9  ;; C%$7 M ++M: %** '')@)@-BUBUVXBY &--aAq9  6;;B @[@[\ #{{=9)+.  r7r&)rLrMrNr%rrrrrrr+rrrr.rwrrrHrrFrOrPs@r6rrs )+,)* 8':9 "\\9 5:\\9 MSTfMg9 + +9 9 r7rceZdZUeed<dZdeffd Zee dde jdzde jdzde jdzde dzd e jdzd edzd e jdzd eed eezfdZxZS)Glm4vTextModelr_)rc  t|||j|_|j|_t j |j|j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_t#||_d|_|j)ycc}w)Nrr_F)r*r+ pad_token_id padding_idx vocab_sizer,rr3 embed_tokensrrnum_hidden_layersrlayersr(r normr( rotary_embrrris r6r+zGlm4vTextModel.__init__As  !.. ++LL):):FO   --i8M  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )..q!R8??=CVCVWXCY[]^L   ! #' 299!\=O=OPQ=RTVWL    !l&8&8&;q&@ ,Q '+L!% kk),,.-  )7;7 % "oom,oW![[ *M)*. /-$7 M*M * -0 &++  r7)NNNNNNN)rLrMrNr$rrr+rrr.rtrwr r}rerrrHrrFrOrPs@r6rr<s   .2.204(,26!%26Q ##d*Q  t+Q &&- Q  Q ((4/ Q $;Q ((4/Q -.Q  ( (Q Q r7rceZdZUdZiZdZeed<ddgZfdZ dZ dZ dd e jd zd e jd zd e jd zd e jd zdee je jff dZee dde j&d e jd zdeedeezfdZee dde j&d e jd zdeedeezfdZ d d e jde j&de j&d zde j&d zfdZee d!d e jd zd e jd zde jd zded zde j&d zde jd zde j&d zd e jd zd e jd zde jd zde jd zdeedeezfdZxZS)" Glm4vModelrFr_rrct||tj|j|_t j|j|_d|_ |jyra) r*r+r _from_config vision_configvisualr text_configlanguage_modelrrrs r6r+zGlm4vModel.__init__sU  &33F4H4HI ,99&:L:LM r7c6|jjSra)rget_input_embeddingsrJs r6rzGlm4vModel.get_input_embeddingss""7799r7c:|jj|yra)rset_input_embeddingsr2rs r6rzGlm4vModel.set_input_embeddingss 007r7Nrimage_grid_thwvideo_grid_thwrrhc F|jjj}|jj}|jj}|jj }g} |h||c|} |t j| }t jd|jd|jd|j|j} d\} } d}|j| j}t| D]\}}|||dk(}|j}g}d}|D]T}||k(rd}n||k(rd}||k(r|s|jd+||k(r|r|jd D|jd Vg}t!j"t|d D]7\}}t%|}|dd}|d ddz}|j|||f9g}d}|D]:\}}}t'|dkDr|d j)dznd}|dk(rQ|| d|| d|| d }!} }|j+| j+|z|!j+|z}$}#}"t j,|"j/d dj1d |#|$zj3}%t j,|#j/dd dj1|"d |$j3}&t j,|$j/ddd j1|"|#d j3}'|jt j4|%|&|'g|z| dz } d}|d k(rb||| d|| d }!} }|| j+|z|!j+|z}$}#}"t7|"D]}(t j8|(j/d dj1d |#|$zj3}%t j,|#j/dd dj1dd |$j3}&t j,|$j/ddd j1d|#d j3}'|jt j4|%|&|'g|z|dz }||| dk\r| dz } d}|dz }||z })|jt j,|)j/dd j1dd |zd}=t j:|dj=dd }*|*j| j| d|||dk(f<| j|*j)dzt'| |z t j8| |jj?d} | | fS||jAjCd dz } | jE|dk(d| j?dj1dd d j|j} | j)dddj)d dd}+|+dz|jd z } | | fSt j,|jd|jj/ddd j1d|jdd } t jF|jddg|j|j} | | fS)aU Calculate the 3D rope index based on image and video's temporal, height and width in LLM. Explanation: Each embedding sequence contains vision embedding and text embedding or just contains text embedding. For pure text embedding sequence, the rotary position embedding has no difference with modern LLMs. Examples: input_ids: [T T T T T], here T is for text. temporal position_ids: [0, 1, 2, 3, 4] height position_ids: [0, 1, 2, 3, 4] width position_ids: [0, 1, 2, 3, 4] For vision and text embedding sequence, we calculate 3D rotary position embedding for vision part and 1D rotary position embedding for text part. Examples: Temporal (Time): 3 patches, representing different segments of the video in time. Height: 2 patches, dividing each frame vertically. Width: 2 patches, dividing each frame horizontally. We also have some important parameters: fps (Frames Per Second): The video's frame rate, set to 1. This means one frame is processed each second. tokens_per_second: This is a crucial parameter. It dictates how many "time-steps" or "temporal tokens" are conceptually packed into a one-second interval of the video. In this case, we have 25 tokens per second. So each second of the video will be represented with 25 separate time points. It essentially defines the temporal granularity. temporal_patch_size: The number of frames that compose one temporal patch. Here, it's 2 frames. interval: The step size for the temporal position IDs, calculated as tokens_per_second * temporal_patch_size / fps. In this case, 25 * 2 / 1 = 50. This means that each temporal patch will be have a difference of 50 in the temporal position IDs. input_ids: [V V V V V V V V V V V V T T T T T], here V is for vision. vision temporal position_ids: [0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100] vision height position_ids: [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1] vision width position_ids: [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1] text temporal position_ids: [101, 102, 103, 104, 105] text height position_ids: [101, 102, 103, 104, 105] text width position_ids: [101, 102, 103, 104, 105] Here we calculate the text start position_ids as the max vision position_ids plus 1. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Returns: position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`) mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`) rrr"r=r)rrFTrrrc |dS)Nr"r)rs r6z+Glm4vModel.get_rope_index.."s [\]^[_r7r:r9r.rr;r)$r_rrimage_token_idvideo_start_token_idvideo_end_token_idr. ones_liker/rIr=rr>rNrr itertoolsgroupbyrrritemrrtrrVrrrrrrrr masked_fill_zeros),r2rrrrrrrrmrope_position_deltastotal_input_idsrI image_index video_indexvideo_group_indexr input_tokensinput_token_typevideo_check_flgtokeninput_type_grouprgroup start_index end_indexllm_pos_ids_listvideo_frame_num modality_type start_idxend_idxst_idxrrr llm_grid_t llm_grid_h llm_grid_wt_indexh_indexw_indext_idxtext_len llm_positionsmax_position_idss, r6get_rope_indexzGlm4vModel.get_rope_indexsLv"[[66II33#{{??![[;; "  n&@ND^'O%!&!A ::""oo '' L(, $K ! +../E/EFN )/ :W ` 9%nQ&71&<= (//1 #% "') 8E 44*."44*/.(//8.0_(//8(//7 8$& "+"3"3I>N4OQ_"`KJC KE"'(1+K %b ! q 0I$++S+y,IJ K $& "#9I7,5M9g?BCS?TWX?X-b1557!;^_F$/*;7:*;7:*;7: 1 FFHFFH(::FFH(::1;J #(,,z":"?"?A"F"M"MbR\_iRi"j"r"r"t"',,z":"?"?2q"I"P"PQ[]_ak"l"t"t"v"',,z":"?"?1b"I"P"PQ[]gik"l"t"t"v(// Wgw&>r1&E&L&LRQ[^hQh&i&q&q&sG&+ll:&>&C&CAr1&M&T&TUVXZ\f&g&o&o&qG&+ll:&>&C&CAq"&M&T&TUVXbdf&g&o&o&qG,33EKK'SZ@[4\_e4ef g*Q.),{0KA0NN'1,K01-'1,$+Y#6(// X0F0K0KAr0R0Y0YZ[]_0`ci0ij*+o7,r!& *: B J J1b Q ?L?O?OP\PcPc?d S!^A%6!%;;<%,,]->->-@1-Ds?[\K]G^-^_oW `p%*LL1FyO_O_$`$j$jkl$m !!66 6)-224;;B?!C )).A*=qA+55a8??2rJMMnNcNcd #/#3#3Au#3#Ea#H#L#LRY]#L#^_`#a (81(<~?S?STV?W(W% !66 6LL!3I#?#?M '3$r7 pixel_valuesc <|j|jj}|j|f|dd|}|jd|jjdzzj }t j|j|}||_|S)T pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. Tr$r:r9) rGrr=r&rrr.rr)r2r.rrr*r+ image_embedss r6get_image_featureszGlm4vModel.get_image_featuress$(():):; $\gNX\g`fg%**2.$++2P2PRS2SS[[] {{>#?#?M '3$r7rimage_featuresvideo_featurescT|||jtj|jjtj |j k(}|jd}||jtj|jjtj |j k(}|jd}n2||jjk(}||jjk(}|j}|jdj|j|j }|At||j|jk(d|d|jd|j}|jdj|j|j }|At||j|jk(d|d|jd||fS)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. rr:z6Image features and image tokens do not match, tokens: z , features: rz6Video features and video tokens do not match, tokens: )rr.rr_rrrallvideo_token_idsumr expand_asr>rnumelrI) r2rrr3r4special_image_maskspecial_video_maskn_image_tokensn_video_tokenss r6get_placeholder_maskzGlm4vModel.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; "+dkk.H.H!H !*dkk.H.H!H +//1/99"=GG VYYZgZnZno  % "01779^=Q=Q=SSHHXXdeseyeyz{e|d}~  ,//1/99"=GG VYYZgZnZno  % "01779^=Q=Q=SSHHXXdeseyeyz{e|d}~ "#555r7rIrjrrkc Z|du|duz r td||j|}||j||dj} t j | dj |j|j} |j||| \}}|j|| }||j|| dj}t j |dj |j|j}|j|||\}}|j||}|t|ts|n|d }||jd k(rtt j|dddfd d }|jj r?|t j"|jj$z }d|z j'}t)xr2|duxr|j*d d k7xs|duxr|j*d d k7}t) xr)| duxr| ddk(xs|duxs|j-dk(}|s|s |j. |j1||| |\}} | |_n|j*\}}}| +| d|j.zj |jnd}t j2||j}|j5d dj7|d}| #|j9||j*dzd}|j;|}|j=dj7ddd}|j>dd||||| d| }tA|jB|jD|jF|jH|j.S)a image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. NrT)r%rr)r3)r4full_attentionrr"r9)dim1dim2r~)rrr:r)rrIrrjrrk)rrjrCrrr)%rrr2rr.rr>rr=r?masked_scatterr-rdictrdiagonalis_floating_pointfinfominrrrIrrr rrtrrYaddrrrrrjrCr)r2rrrIrjrr.r!rrrrkrr1 image_maskrr, video_maskattention_mask_tensorprefill_compiled_stageprefill_noncompiled_stage batch_sizerdeltaoutputss r6rFzGlm4vModel.forwards2  -t"; <YZ Z  7D557 BM  #22<]a2bppL 99\q9<<]=Q=QS`SfSfgL 55i_k5lMJ)88\RM  *223Fdh2iwwL 99\q9<<]=Q=QS`SfSfgL 55i_k5lMAz)88\RM  &0&FN[kLl "%05J5O5OST5T(-7LQPQT7RYZab(c%(..@@,AEKKPePkPkDlDpDp,p)-03H-H,M,M,O) &>%?&$&B9??1+=+BO!-M-2E2Ea2HA2M #-E,F(F)t+Fq0AQ0FV#t+T/M/M/OST/T &'*CHXHXH`,0,?,?""#8 -@-) k $/ -:,?,?) J&1$A&)9)99==m>R>RS %||J}?S?ST +00B7>>z2N !-!33J%++a.4PVW3XE+//6 +55a8??2rJ %$%% %)+')   (%77#33!//))((   r7rrrar) NNNNNNNNNNN)rLrMrNr_checkpoint_conversion_mappingaccepts_loss_kwargsr#rrr+rrr.rtrwrHr rrr}rrrr-r2r?r rrFrOrPs@r6rrs"%'" 02DE:8 .22626.2 |7##d*|7((4/|7((4/ |7  t+ |7 u||U\\) * |7|37"..((4/+,  + + 837''((4/+,  + + 04837 (6##(6(((6))D0 (6 ))D0 (6T.2.204(,26,08<2626/326g ##d*g  t+g &&- g  g ((4/ g llT)g #..5g ((4/g ((4/g %%,g ((4/g +,g  ) )g g r7rzQ Base class for Glm4v causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZejdzed<y) Glm4vCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. NlosslogitsrjrCrr)rLrMrNrsrWr.r}rrXrjr rCrHrrrtrr7r6rVrVGs &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/6+/K!!D(/r7rVc eZdZiZddiZdZfdZdZdZe dde jd e jdzd e ed eezfd Ze dd e jde jdzd e ed eezfdZee dde jdzde j(dzde jdzdedzde jdzde jdzd e j(dzde jdzde jdzd e jdzde jdzdee j(zd e ed eezfdZ d fd Z dde jdzde j(dzd ee j(e j(ffdZ d!dedede jdzd ee jeeefffdZxZ S)"Glm4vForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightFct||t||_t j |j j|j jd|_ |jy)NFrV) r*r+rrr,rYrr3rlm_headrrs r6r+z&Glm4vForConditionalGeneration.__init__jsS  ' yy!3!3!?!?ASASA^A^ejk  r7c6|jjSra)rrrJs r6rz2Glm4vForConditionalGeneration.get_input_embeddingsqszz..00r7c:|jj|yra)rrrs r6rz2Glm4vForConditionalGeneration.set_input_embeddingsts ''.r7Nr!rrrhc @|jjd||d|S)r#)r!rr)rr-)r2r!rrs r6r-z0Glm4vForConditionalGeneration.get_video_featuresws/-tzz,, 3N V\  r7r.rc @|jjd||d|S)r0)r.rr)rr2)r2r.rrs r6r2z0Glm4vForConditionalGeneration.get_image_featuress'-tzz,,p,Wepioppr7rrrIrjrlabelsrklogits_to_keepc  |jd|||| | ||||| d | }|d}t| tr t| dn| }|j |dd|ddf}d}|2|j |||j jj}t|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Glm4vForConditionalGeneration >>> model = Glm4vForConditionalGeneration.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```) rr.r!rrrIrrjrrkrN)rXrar)rWrXrjrCrrr)rrrslicer\ loss_functionr_rrrVrjrCrr)r2rrrIrjrrar.r!rrrkrbrrRrC slice_indicesrXrWs r6rFz%Glm4vForConditionalGeneration.forwardsx$**  % 3))%)+')      9C>SV8W~ot4]k mA}a,?@A  %%VFt{{OfOfOqOq%rD*#33!//))++   r7c  ft||f||||||| | | || d | }d|d<| s |r d|d<d|d<|S)N) rjrrrkrIr.r!rrris_first_iterationrIr.r!)r*prepare_inputs_for_generation)r2rrjrrrkrIrr.r!rrrhr model_inputsr5s r6riz;Glm4vForConditionalGeneration.prepare_inputs_for_generationss$w<  +)')%% 3))1  "(, ^$!i+/L (26L. /r7c|||jtj|jjtj |j k(d}||jtj|jjtj |j k(d}||jtj|jjtj |j k(d}nK||jjk(}||jjk(}||jjk(}tj|j|jz d}|dkD}||z}|jd} |jd} | | fS)aa Get the number of images and videos for each sample to calculate the separation length of the sample tensor. These parameters are not passed through the processor to avoid unpredictable impacts from interface modifications. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Returns: image_nums (`torch.LongTensor` of shape `(batch_size, num_images_sample)`) video_nums (`torch.LongTensor` of shape `(batch_size, num_videos_sample)`) r).rr"rr) rr.rr_image_start_token_idrrrrrrr8) r2rris_imageis_video_start is_video_end video_level inside_videostandalone_images image_counts video_countss r6_get_image_nums_and_video_numsz#5#5#7,:J:J:L#LRST "Q % 6),,,3 %))a)0 \))r7 expand_sizeis_encoder_decoderc dk(rfSgdfd}fd}|jd||r*jd td|dd<fS) Nr")r.rr!rsecond_per_grid_tsc4 jdd} jdd} j jdd\}}d}|D]9}|dk(rctj|t |}|Dcgc]'}tj |dj )} }|||| ||<l|dk(rt |} |||| ||<|d k(rctj|t |}|Dcgc]'}tj |dj )} }|||| ||<|dk(rt |} |||| ||<|d k(s |||t | ||<<|Scc}wcc}w) Nrrr)rctj||}|gdg|jdz zz}tj|Dcgc]}|j|c}d}|Scc}w)Nr"rr)r.rr{rr)rr repeat_timessamples repeat_argssamplerRs r6_repeat_interleave_sampleszGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samplesosa++a1+nsaeegk/BB g#VFMFMM;$?#V\]^ $WsA&r.r"r)rr|r!ry)r7rur.rrr&r8)dict_to_expandrr image_nums video_numsrrr}rrrvr model_kwargsr2s r6"_expand_dict_for_generation_visualzgGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visualhs)--.>EN)--.>EN%)%H%H)9)9/4)P&I& "J   & .(#kk.$z:JKGMTU6uzz&a8<<>UGU*D&s+W;+N3',,":.G*D&s+W;+N3'11#kk.$z:JKGMTU6uzz&a8<<>UGU*D&s+W;+N3',,":.G*D&s+W;+N3'00*D&s+T*5ET_+N3'7 <" !3VVs =,F,Fc|D]J}|dk7s ||t||tjs-|vs2||jd||<L|S)Nrkrr)rr.rwrY)rrrv visual_keyss r6_expand_dict_for_generationz`Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generationsl% d++&s+7">##6 E;.*8*=*O*OP[ab*O*cN3' d" !r7rrencoder_outputszMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)rYr7r)r2rvrwrrrrrs`` `` @r6_expand_inputs_for_generationz;Glm4vForConditionalGeneration._expand_inputs_for_generationWs ! l* *w + "Z ":,G  !33KQ3GI2<@  12: !pqq.I,WhJi.jL* +,&&r7ra) NNNNNNNNNNNr) NNNNNTNNNNF)r"FN)!rLrMrNrS_tied_weights_keysrTr+rrrr.r}rtrrrHrr-r2rrwr rrVrFrirurerErrrrOrPs@r6rZrZds%'"*,VW1/37 ".. ((4/ +,  + +   37 q'' q((4/ q+, q + + q q.2.204(,26*.,08<262626-.Y ##d*Y  t+Y &&- Y  Y ((4/ Y   4'Y llT)Y #..5Y ((4/Y ((4/Y ((4/Y ell*Y +,Y  , ,Y Y |  )\.26*##d*6*||d*6* u||U\\) * 6*t#(-1 U'U'!U'##d* U' uc3h/ 0 U'r7rZ)rZrrrr)r)r")Zrcollections.abcr dataclassesrtypingrrr.torch.nnr,torch.nn.functionalrrrr r activationsr cache_utilsr r generationr integrationsr masking_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrr r!configuration_glm4vr#r$r%Moduler(rRrgryrrrrwrHrrrrrrrr(rWr`rbrvrrrrrrrVrZ__all__rr7r6rs($! &!.)7/B9``KF&^]PPY'J299J(J( fBII fBII(   fRYYf"HBIIHV(  ||   +0<< >Cll  5<< %&  UU\\ U# U%,, U&% II% <<% % << % LL4' %  %%'(%4P299Pf16JryyJZ6%PE)E)P)299)$161h  0{0  0$2?220} +} @g )g g T\ %\ \ ~   0+0  0.H'$8/H'V xr7