i\kdZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z dd lmZdd lmZdd lmZdd lmZmZdd lmZddlmZmZmZddlmZddlm Z er ddl!m"Z"ddl#m$Z$ejJe&Z'de(de(dejRfdZ*dejRdejRfdZ+dejRdejRdejRdejRfdZ,GddejZZ.Gd d!ejZZ/Gd"d#eZ0eGd$d%eZ1eGd&d'e1Z2ed()Gd*d+e1eZ3gd,Z4y)-zPyTorch CodeGen model.N)Union)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)auto_docstringis_torch_flex_attn_availablelogging)is_flash_attention_requested) CodeGenConfig) BlockMask)make_flex_block_causal_masknum_posdimreturncddtjd|dtj|z zz }tjdtj|tjj |j }tj tj |tj|fdS) Ng?i'r)dtypez i , j -> i jrr)torcharangeint64einsumfloatcatsincos)rrinv_freq sinusoid_inps v/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/codegen/modeling_codegen.pycreate_sinusoidal_positionsr*1se QQekk JS PQRH<< WEKK0X0^0^0`bjkqqsL 99eii -uyy/FGQ OOxc|dddddddddf}|dddddddddf}tj| |fd}|jdS)Nrrr)rstackflatten)r,x1x2s r)rotate_every_twor48sS 1aCaC<B 1aADqD= B bS"I2&A 99R=r+tensorr%r&ctj|dddddddfdd}tj|dddddddfdd}||zt||zzS)Nrr)rrepeat_interleaver4)r5r%r&s r)apply_rotary_pos_embr8@s^  ! !#aD!m"4a ;C  ! !#aD!m"4a ;C SL-f5; <>"#;N$NOr+ct|jdk(r$|jdddddj}n\t|jdk(r#|jddddj}n!t dt|j|j dd ||zfz}|j |S) zM Merges attn_head_size dim and num_attn_heads dim into n_ctx rrrrz3Input tensor rank should be one of [4, 5], but is: Nr/)lenr_permute contiguousrQsizeview)r[r5rOattn_head_size new_shapes r) _merge_headszCodeGenAttention._merge_headsls v||  !^^Aq!Q2==?F  ! #^^Aq!Q/:: MMD$<$<<F !KK 9"Euc!!%)A)A4==Y_!`T%=%=t}}U[\!!%)A)A4==Y_!` aAq)..  ! !\%8%8 8-001D1DEO#2D  .;;vv||B'71'<"ES ?? &1a!24??!223EAq$//"334F!Q#4T__#445E1aDOO$556F(S9E(S9E))UFO4CIIufo26E&sC5C(S9Ekk!Q1% aAq)  !)-"0 L $**366-2E2E+Ft~~_klJC%)JJuc5.$Q! \'' T5M5Mt}}] mmK0 ((5 L((r+NNNNFFN)rL __module__ __qualname__r@rdror}r FloatTensorr LongTensorbooltupleTensorr __classcell__rKs@r)r:r:Fs @ &$ ):$(3704!&).26G)((4/G)DLG)))D0 G) &&- G) $; G) $;G)((4/G) ellE%,,//0  eELL15s9J3KK L M  G)r+r:c\eZdZfdZdej dzdej fdZxZS) CodeGenMLPct||j}tj|||_tj|||_t|j|_ tj|j|_ yr) r?r@n_embdrrUfc_infc_outractivation_functionactrCrFdropout)r[intermediate_sizer\rNrKs r)r@zCodeGenMLP.__init__se MM YYy*;< ii 19= &445zz&"4"45 r+r~Nrc|j|}|j|}|j|}|j|}|Sr)rrrr)r[r~s r)rzCodeGenMLP.forwards@ =1 /  M2  ]3 r+)rLrrr@rrrrrs@r)rrs,6U%6%6%=%BSBSr+rc2eZdZd fd Z ddej dzdedzdej dzdejdzdedzdedzd ejdzd e eje eje ej d ffzdzfd Z xZ S) CodeGenBlockNct||j |jnd|jz}t j |j|j |_t|||_ t|||_ y)Nrgeps) r?r@n_innerrr LayerNormlayer_norm_epsilonln_1r:attnrmlp)r[r\rH inner_dimrKs r)r@zCodeGenBlock.__init__sc &,nn&@FNNa&--FW LLF4M4MN $VY7 i0r+r~rryrrrrr.c |}|j|}|j|||||||\} } |j|} | | z|z}|| fS)N)r~rryrrrr)rrr) r[r~rryrrrrresidual attn_outputsrzfeed_forward_hidden_statess r)rzCodeGenBlock.forwardsp! -0 %)YY'!)%/)&/& " l&*XXm%<"$'AAHL l**r+rr) rLrrr@rrrrrrrrrrs@r)rrs1$(3704!&).26+((4/+DL+))D0 + &&- + $; + $;+((4/+ u|| uU\\59J9JC9O3P%PQ QTX X+r+rc@eZdZUeed<dZdZdgZdZdZ fdZ xZ S)CodeGenPreTrainedModelr\ transformerTrpast_key_valuesct||t|tr?t j |j t|j|jyyr) r? _init_weights isinstancer:initcopy_r=r*rBrY)r[modulerKs r)rz$CodeGenPreTrainedModel._init_weightssI f% f. / JJv--/J6K_K_agatat/u v 0r+) rLrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraphrrrs@r)rrs6 %&*#'("3!wwr+rceZdZfdZdZdZe ddejdzde dzdejdzdejdzd ejdzd ejdzd e dzd e dzd e dzde dzdejdzde e zfdZ ddeej dfdej dej de d e f dZedej dededej(dej def dZxZS) CodeGenModelc t|||j|_|j|_t j |j|j|_t j|j|_ t jt|jDcgc]}t||c}|_t j |j|j"|_t'|j(|j*|j,z|_d|_|j1ycc}w)N)rHrF)r?r@rrN vocab_sizer EmbeddingwterC embd_pdropdrop ModuleListrangen_layerrhrrln_fminrXn_ctxrOgradient_checkpointing post_init)r[r\irKs r)r@zCodeGenModel.__init__'s   ++<< 1 14>>BJJv001 5QWQ_Q_K`aa Vq AabLLV5N5NO f//A[A[1[\&+#  bs,Ec|jSrr)r[s r)get_input_embeddingsz!CodeGenModel.get_input_embeddings7s xxr+c||_yrr)r[new_embeddingss r)set_input_embeddingsz!CodeGenModel.set_input_embeddings:s !r+N input_idsrrytoken_type_idsr inputs_embedsrroutput_hidden_states return_dictrrc ||n|jj}| | n|jj} ||n|jj}| | n|jj} |du|duz r t d|j r%|jr|rtjdd}||j|}|r|t|j}|jd} | 9||jnd}tj||| z|j } || j#d}|j%||| ||}|}|(|j'd | }|j|}||z}|j)|}d | |j+d f}|rd nd}| rd nd}t-|j.D]-\}}| r||fz}|||||||| }|d}|s%||dfz}/|j1|}|j'|}| r||fz}| st3d ||||fDSt5|||| S)a inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r\rrrr.)rryrrrrc3&K|] }|| ywrr).0vs r) z'CodeGenModel.forward..sghgts)last_hidden_staterr~ attentions)r\rrruse_return_dictrQrtrainingrIrJrr r_get_seq_lengthrr r unsqueeze_update_causal_maskrlrrk enumeraterrrr )r[rrryrrrrrrrrkwargs seq_lengthpast_seen_tokensr{r~token_type_embeds output_shapeall_self_attentionsall_hidden_statesrblockoutputss r)rzCodeGenModel.forward=s,2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKO"((+  !CRC^==?de "\\*:?L] &  %+00Z@N $ 8 ),==M -0 J (:(:2(>? $5b4"6BD!$&&) JHAu#$58H$H!**)#"3-G$AJM &9WQZM&I#! J$ -0 %**<8  1]4D D )?T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCKQZ[Kr+rrrrc ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nrg) fill_valuerrr)diagonalrr.r)rrrrfullrtriur r^expandcloner_rq masked_fill) ryrrrrrrr{r mask_length padding_masks r)rzBCodeGenModel._prepare_4d_causal_attention_mask_with_cache_positions>  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r+) NNNNNNNNNNN)F)rLrrr@rrrrrrrrrr rrrr staticmethodintrrrrs@r)rr%s ".2(,37260426!%)-,0#'26g ##d*g g ))D0 g ((4/ g &&- g ((4/g $;g  $;g #Tkg D[g ((4/g  ( (g g `#( BellK78BllB B  B  BH4 444{{ 4  4  44r+rzM The CodeGen Model transformer with a language modeling head on top. ) custom_introceZdZddiZfdZe ddejdzdedzdejdzdejdzd ejdzd ejdzd ejdzd e dzd e dzde dzde dzdejdzde ejzde ezfdZxZS)CodeGenForCausalLMzlm_head.weightztransformer.wte.weightct||t||_t j |j |j|_|jyr) r?r@rrrrUrrlm_headr)r[r\rKs r)r@zCodeGenForCausalLM.__init__-sE  '/yy0A0AB  r+Nrrryrrrlabelsrrrrrlogits_to_keeprc | | n|jj} |j|||||||| | | |  }|d}t| tr t | dn| }|j |dd|ddf}d}|*|jd|||jjd|}| s|f|ddz}||f|zS|St|||j|j|jS)aG inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) rryrrrrrrrrr)logitsr&rr)lossr)rr~rr) r\rrrr slicer% loss_functionrrrr~r)r[rrryrrrr&rrrrrr'rtransformer_outputsr~ slice_indicesr)r*outputs r)rzCodeGenForCausalLM.forward5s"8&1%SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopDY!4QR!88F)-)9TGf$ Ev E%/??-;;*55   r+) NNNNNNNNNNNNr)rLrr_tied_weights_keysr@rrrrrrr rrrrrrs@r)r#r#%sM +,DE.2(,37260426*.!%)-,0#'26-.> ##d*> > ))D0 > ((4/ > &&- > ((4/>   4'> $;>  $;> #Tk> D[> ((4/> ell*> ' '!> > r+r#)r#rr)5__doc__rRtypingrrrrr activationsr cache_utilsrr generationr modeling_attn_mask_utilsr modeling_layersr modeling_outputsr rmodeling_utilsrutilsrrr utils.genericrconfiguration_codegenr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerrLrIr rr*r4r8Moduler:rrrrr#__all__rr+r)rCsv  &!.)>9O- :0 !;J   H %PP3P5<<P ==ELL=u||=X]XdXd= T)ryyT)p(!+-!+H w_ w w|)||~ J /J  J Z Kr+