iA dZddlZddlmZddlZddlmZddlmZmZm Z m Z ddlm Z ddl mZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZddlm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&e!r ddl'm(Z(ddl)m*Z*e"jVe,Z-dej\de/dej`dej\fdZ1dej\dej\de2de3dej\f dZ4dej\dej\fdZ5dej\dej\dej\fd Z6Gd!d"ejnjpZ9Gd#d$ejtZ;Gd%d&ejtZ<Gd'd(ejtZ=Gd)d*eZ>e Gd+d,eZ?e Gd-d.e?Z@e d/0Gd1d2e?eZAe d30Gd4d5e?ZBe Gd6d7e?ZCe Gd8d9e?ZDgd:ZEy);zPyTorch BLOOM model.N)Union)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)Cache DynamicCache StaticCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringis_torch_flex_attn_availablelogging)is_flash_attention_requested) BloomConfig) BlockMask)make_flex_block_causal_maskattention_mask num_headsdtypereturnc|j\}}dtjtj|z}t j ddtj|dz z z|j tj}t jdd|z|j tj}t j||}||k7rt j ddtjd|zdz z z|j tj} t|||z } t jddd| zzd|j tj} t j|t j| | gd}|jddz |zdddddf} |d | z} | j||zd|j|S) a Link to paper: https://huggingface.co/papers/2108.12409 Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation: with l being a tensor, and a fixed value `softmax(l+a) = softmax(l)`. Based on https://github.com/ofirpress/attention_with_linear_biases/blob/a35aaca144e0eb6b789dfcb46784c4b8e31b7983/fairseq/models/transformer.py#L742 TODO @thomasw21 this doesn't work as nicely due to the masking strategy, and so masking varies slightly. Args: Returns tensor shaped (batch_size * num_heads, 1, max_seq_len) attention_mask (`torch.Tensor`): Token-wise attention mask, this should be of shape (batch_size, max_seq_len). num_heads (`int`): number of heads dtype (`torch.dtype`, *optional*, default=`torch.bfloat16`): dtype of the output tensor r devicer!rrdimN).N)shapemathfloorlog2torchtensorr&float32arangeint32powmincatcumsumreshapeto)rr r! batch_size seq_lengthclosest_power_of_2basepowersslopes extra_basenum_remaining_heads extra_powers arange_tensoralibis r/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/bloom/modeling_bloom.pybuild_alibi_tensorrE6s",11J djj9)=>> << tyy!34q899:;NDYDYafanan D\\!Q!33NI1dTU:VM 9  -E ==i/J ? B B5 IIxresidualprobtrainingc@tj|||}||z}|S)a  Dropout add function Args: x (`torch.tensor`): input tensor residual (`torch.tensor`): residual tensor prob (`float`): dropout probability training (`bool`): training mode )prJ)Fdropout)rGrHrIrJouts rD dropout_addrPbs$ ))A 1C S.C JrFc\|dzdtjd|zdd|z|zzzzzS)z Custom bias GELU function. Adapted from Megatron-DeepSpeed code. Here we use a simple implementation (inference) to make the model jitable. Args: x (`torch.tensor`): input hidden states ??e3E?rHm?r.tanh)rGs rDbloom_gelu_forwardrXus8 s7cEJJzA~X\A=M9M'NOO PPrFgc|d}tjd|zdd|z|zzz}d|zd||zz dd|z|zzzzdd|zzz}||zS)a  gradient of tanh approximation of gelu gradient of actual gelu is: 0.5 * (1. + torch.erf(x * 0.70710678)) + 0.3989423 * x * torch.exp(-0.5 * x * x) Args: g (`torch.tensor`): gradient output tensor x (`torch.tensor`): input tensor rrTrrUrRg6vf?rV)rYrGtanh_outffs rDbloom_gelu_backr]sz !Azz*q.A1 q0@,@ABH qQH,,lQ>NQR>R1RS TWZ^_bj^jWk kB 6MrFceZdZedej dej fdZedej dej fdZy) GeLUFunctioninputr"c:|j|t|SN)save_for_backwardrX)ctxr`s rDforwardzGeLUFunction.forwards e$!%((rF grad_outputc6|j}t||}|Srb) saved_tensorsr])rdrfr`tmps rDbackwardzGeLUFunction.backwards!!k51 rFN)__name__ __module__ __qualname__ staticmethodr.TensorrerjrFrDr_r_sT)ELL)U\\))5<<ELLrFr_cZeZdZdZfdZdej dej fdZxZS) BloomGeluzN Partly copied from Megatron-DeepSpeed code and adapted for our needs c"t|yrb)super__init__)self __class__s rDruzBloomGelu.__init__s rFrGr"c,tj|Srb)r_apply)rvrGs rDrezBloomGelu.forwards!!!$$rF) rkrlrm__doc__rur.rore __classcell__rws@rDrrrrs(%%%,,%rFrrcveZdZddededzffd ZdejdeejejejffdZ dejdejfd Z dd ejd ejd ejd ejde dzde de dejdzfdZxZS)BloomAttentionNconfig layer_idxct||j|_|j|_|j|_|j |_|j|j z|_|j|_|j|_ |j|j z|jk7r&td|jd|j ddtj|jz |_ d|_||_|-t j#d|j$j&dt)j*|jd|jzd |_t)j*|j|j|_t)j0|j2|_y) NzA`hidden_size` must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).rSzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r Tbias)rtrupretraining_tpslow_but_exact hidden_sizen_headr head_dim split_sizehidden_dropout ValueErrorr+sqrtinv_norm_factorbetarlogger warning_oncerwrkrLinearquery_key_valuedenseDropoutattention_dropout)rvrrrws rDruzBloomAttention.__init__sz $33$33!--((DNN: **$33 ==4>> )T-=-= =STXTdTdSefNN#2'  #TYYt}}%== "      !8!8 9:,,  "yy)9)91t?O?O;OVZ[YYt//1A1AB !#F,D,D!ErF fused_qkvr"c|j\}}}|j|||jd|j}|ddddfj dd}|ddddfj dd}|ddddfj dd}|||fS)a Split the last dimension into (num_heads, head_dim) and reshapes to (bs, heads, len, dim) shape without making any copies, results share same memory storage as `fused_qkv` Args: fused_qkv (`torch.tensor`): [batch_size, seq_length, num_heads * 3 * head_dim] Returns: query: [batch_size, num_heads, seq_length, head_dim] key: [batch_size, num_heads, seq_length, head_dim] value: [batch_size, num_heads, seq_length, head_dim] r .rNrr$)r*viewr r transpose)rvrr9r:three_times_hidden_size query_layer key_layer value_layers rD_reshapezBloomAttention._reshapes;D//7 J 7NN:z4>>1dmm\ Q *44Q: c1ai(221a8 Q *44Q: I{22rFrGc |j\}}}||jz}|j||j||j}|j dddd}|j |||j|jzS)z Merge heads together over the last dimension Args: x (`torch.tensor`): [batch_size * num_heads, seq_length, head_dim] Returns: torch.tensor: [batch_size, seq_length, num_heads * head_dim] rr$rr )r*r rrpermuter7)rvrGbatch_size_and_num_headsr:_r9s rD _merge_headszBloomAttention._merge_headssy34''/ *a-?  FF:t~~z4== I IIaAq !yyZ$--1OPPrF hidden_statesrHrCr layer_past use_cacheoutput_attentionscache_positionc |j\} } } |j|} |j| \} }}|%d|i}|j|||j|\}}| j | |j zd|j} |j | |j zd|jjdd}|j | |j zd|j}|j| ||j|j}|j| |j | d}|#|ddddddd|jdf}||z}tj|dtj j#| j$}|j'|}|j| |j z| d}tj(||}|j+|}|j,dkDr|j.r|j0|j,z }tj2|}t5|j,D]z}|tj6|ddddt9||zt9|dz|zf|j:j<ddt9||zt9|dz|zfz}|n|j;|}t?|||j@|jB}||fS)Nrr))batch1batch2ralpha)r(r!r)"r*rrupdaterr7r rrbaddbmmrrrrMsoftmaxr.r0r8r!rbmmrrrr zeros_likerangelinearintrweightrPrrJ)rvrrHrCrrrrrr9q_lengthrrrrr cache_kwargsattention_scores attn_weights causal_maskattention_probsattention_probs_reshaped context_layerslices output_tensoris rDrezBloomAttention.forwards#0"5"5 Ha((7 .2mmI.F+ Y  !,n=L%/%6%6y+t~~_k%l "I{"))*t~~*Er4==Y %%j4>>&A2t}}U__`bdfg !))*t~~*Er4==Y !==&& ) (,,ZSUV  %(Aq2GIOOB4G2G)GHK'+5L))Lb NQQR]RcRcd00A$3#7#7 T^^8SU]_a#b  ":KH ))-8     "t':':%%(;(;;F!,,];M4../  -!!QAJ#q1u>N:O(O"OPJJ%%aQZ3A?O;P)P&PQ1!   !JJ}5M#M8T=P=PRVR_R_` o--rFrbNFFN)rkrlrmrrrur.rotuplerrr bool LongTensorrer{r|s@rDr~r~sF{FsTzFB3%,,35u||UZUaUa9a3b3(QellQu||Q>$("'26C.||C.,,C.|| C.  C. DL C.C. C.((4/C.rFr~cteZdZdeffd Zdej dej dej fdZxZS)BloomMLPrc6t||j}|j|_|j|_t j |d|z|_t|_ t j d|z||_ |j|_ y)N) rtrurrrrr dense_h_to_4hrr gelu_impl dense_4h_to_hr)rvrrrws rDruzBloomMLP.__init__Bsz (( $33$33YY{A OD"YYq; D$33rFrrHr"c |j|j|}|jdkDr|jrt j |}|j jjd|jz }t|jD]z}|tj|ddddt||zt|dz|zf|j jddt||zt|dz|zfz}|n|j |}t|||j|j}|S)Nrr))rrrrr.rrrr*rrMrrrPrrJ)rvrrHintermediate_outputrroutputs rDrezBloomMLP.forwardMs.t'9'9-'HI    "t':':"'"2"28"< ''..44R84;N;NNF4../ &9AHH!!QAJ#q1u>N:O(O"OP&&--aQZ3AQWGWCX1X.XY='#  #'"4"4]"C 0(D8g8g5$33rFrrCrrrrrc |j|}|jr|} n|} |j|| ||||||\} } |j| }|jr|} n| } |j || } | | fS)N)rrrCrrr)rrrrr) rvrrCrrrrrlayernorm_outputrHattention_outputrrs rDrezBloomBlock.forwardos // >  8 8'H$H*.)<)<  !)/)*= * &, 889IJ  8 8'H'H*H5|##rFrbr) rkrlrmrrrur.ror rrrer{r|s@rDrr`s 4{ 4sTz 4&$("'26,$||,$||,$ ,$ DL ,$  ,$ ,$((4/,$rFrc.eZdZUeed<dZdZdgZdZdZ y)BloomPreTrainedModelr transformerTrpast_key_valuesN) rkrlrmr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraphrprFrDrrs( %&*#%"3!rFrcbeZdZdeffd Zdej dedejdej fdZ dZ d ej fd Z e dd ejd zd ed zdej d zdejd zded zded zded zded zdejd zdeej dfezfdZ ddeej dfdej dej d edef dZedej dededejdej def dZxZS) BloomModelrc "t|||j|_|j|_t j|j|j|_ t|j|j|_ t jt|jDcgc]}t!||c}|_t|j|j|_d|_|j)ycc}w)Nr)rF)rtrur embed_dimrr r Embedding vocab_sizeword_embeddingsrrword_embeddings_layernorm ModuleListrnum_hidden_layersrhln_fgradient_checkpointing post_init)rvrrrws rDruzBloomModel.__init__s  ++ "||F,=,=t~~N)24>>vG`G`)a&vOgOgIhiA 6Q ?ijdnn&2K2KL &+#  js.D rr r!r"ct|||Srb)rE)rvrr r!s rDrEzBloomModel.build_alibi_tensors!.)UCCrFc|jSrbr)rvs rDget_input_embeddingszBloomModel.get_input_embeddingss###rFnew_embeddingsc||_yrbrrvrs rDset_input_embeddingszBloomModel.set_input_embeddingss -rFN input_idsr inputs_embedsrroutput_hidden_states return_dictr.c ||n|jj}||n|jj}||n|jj}||n|jj}|du|duz r t d|j r%|jr|rtjdd}||j|}|r|t|j}|j\} } } ||jnd}| |z}| %tj||| z|j } |j#|}|rdnd}|rdnd}|$tj$| |f|j }n|j'|j }|j)||j*|j, }|j/||| ||}t1|j2D]-\}}|r||fz}|||||||| }|d}|s%||d fz}/|j5|}|r||fz}|st7d ||||fDSt9|||| S) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rrr&rp)r!)rrrrrCrrc3&K|] }|| ywrbrp).0vs rD z%BloomModel.forward..(sghgts)last_hidden_staterr attentions)rrrruse_return_dictrrrJrrrr r*get_seq_lengthr.r1r&ronesr8rEr r!_update_causal_mask enumeraterrrr)rvrrrrrrrrrkwargsr9r:r past_lengthseq_length_with_pastrall_self_attentionsall_hidden_statesrCrrblockoutputss rDrezBloomModel.forwards62C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKO$1$7$7! J:I:Uo446[\ )K7  !"\\+{Z7OXeXlXlmN66}E $5b4"6BD  !"ZZ5I(JS`SgSghN+..}/C/CDN''mNaNa'b.. M>?L] "$&&) JHAu#$58H$H!**#"3-G$AJM &9WQZM&I#! J& -0  1]4D D )?T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCKQZ[KrFrrr9c ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr) fill_valuer!r&r)diagonalrr)r)r(r.r*r4fullr&triur1r7expandcloner*r8 masked_fill) rrrr!rr9rrr. mask_length padding_masks rDr(z@BloomModel._prepare_4d_causal_attention_mask_with_cache_positionxs>  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12rF NNNNNNNNN)F)rkrlrmrrur.rorr!rErrrrr rrrrerrrnr(r{r|s@rDrrs{*DD#DV[VaVaDfkfrfrD$.5<<..2(,.215!%)-,0#'26h ##d*h h  t+ h ''$. h $; h  $;h #Tkh D[h ((4/h  u||S !$M Mh h b#( BellK78BllB B  B  BH4 444{{ 4  4  44rFrz The Bloom Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introceZdZddiZdeffd ZdejfdZ dfd Z e dd ejdzd e dzd ejdzd ejdzd ejdzde dzde dzde dzde dzdejdzdeejzdeejezfdZxZS)BloomForCausalLMzlm_head.weightz"transformer.word_embeddings.weightrct||t||_t j |j |jd|_|jyNFr) rtrurrrrrrlm_headrrvrrws rDruzBloomForCausalLM.__init__sI  %f-yy!3!3V5F5FUS  rFrc||_yrb)r?rs rDset_output_embeddingsz&BloomForCausalLM.set_output_embeddingss % rFNc 8t||f||||||d|} t|trq|o|j } |j \} } | | z } t j| | |j|j}t j||gd}|| d<| S)N)rrrrris_first_iterationr%r)r'r) rtprepare_inputs_for_generationr$r r'r*r.zerosr&r!r5)rvrrrrrrrDr model_inputsrr9r:diff new_attn_maskrws rDrEz.BloomForCausalLM.prepare_inputs_for_generationsw<   +)')1     o{ 38R+??AM%3%9%9 "J  :-D!KK DAVAV^l^r^rsM"YY 'FBON-;L) *rFrrrrlabelsrrrrrlogits_to_keepr"c | | n|jj} |j|||||||| |  } | d}t| tr t | dn| }|j |dd|ddf}d}|8|j|||jj| jd}| s|f| ddz}||f|zS|St||| j| j| jS)a\ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)rrrrrrrrrnum_items_in_batch)rrMrlosslogitsrrr )rr rr$rslicer? loss_functionrgetrrrr )rvrrrrrJrrrrrrKrtransformer_outputsr slice_indicesrPrOrs rDrezBloomForCausalLM.forwards+B&1%SV8W~ot4]k mA}a,?@A  %%;;11#)::.B#C &DY!4QR!88F)-)9TGf$ Ev E0/??-;;*55   rF)NNNNTF) NNNNNNNNNNr)rkrlrm_tied_weights_keysrrur.rorBrErrr rrrrrer{r|s@rDr<r<s_+,PQ{&ELL&  #J.2(,.2-1&*!%)-,0#'26-.F ##d*F F  t+ F ||d* F  t# F $;F  $;F #TkF D[F ((4/F ell*F  u|| @ @F F rFr<a The Bloom Model transformer with a sequence classification head on top (linear layer). [`BloomForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZdeffd Ze ddejdzdedzdejdzdejdzdejdzd e dzd e dzd e dzd e dzd e eje zfdZ xZS)BloomForSequenceClassificationrct|||j|_t||_t j |j|jd|_|jyr>) rtru num_labelsrrrrrscorerr@s rDruz'BloomForSequenceClassification.__init__CsV   ++%f-YYv1163D3D5Q  rFNrrrrrJrrrrr"c r| | n|jj} |j|||||||| } | d} |j| } ||jd}n|jd}|jj |dk7r t d|jj d}n|||jj k7j| jtj}tj|jd| jtj}||zjd}n.d}tj|jj d| tj|| j |f}d}|^|jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}nc|||}nY|jj"d k(rt1}|||}n,|jj"d k(rt3}|||}| s|f| ddz}||f|zS|St5||| j6| j8| j: S)6 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r)r%z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationrN)rr rr[r* pad_token_idrr8r&r.r2r1argmaxrrrwrk problem_typerZr!longrrsqueezerrrrrr )rvrrrrrJrrrrrrTrrPr9last_non_pad_token non_pad_mask token_indices pooled_logitsrOloss_fctrs rDrez&BloomForSequenceClassification.forwardLs>&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- v6))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   rFr9)rkrlrmrrurr.rr rorrrrer{r|s@rDrXrX4s{.2(,.2-1&*!%)-,0#'e ##d*e e  t+ e ||d* e  t# e $;e  $;e #Tke D[e  u|| ? ?e e rFrXceZdZdeffd Ze ddejdzdedzdejdzdejdzdejdzd e dzd e dzd e dzd e dzd e eje zfdZ xZS)BloomForTokenClassificationrct|||j|_t||_t |dr|j |j }n't |dr|j |j}nd}tj||_ tj|j|j|_ |jy)Nclassifier_dropoutrg?)rtrurZrrhasattrrorrrrNrr classifierr)rvrrorws rDruz$BloomForTokenClassification.__init__s   ++%f- 6/ 0V5N5N5Z!'!:!:  V- .63H3H3T!'!6!6 !$ zz"45 ))F$6$68I8IJ rFNrrrrrJrrrrr"c | | n|jj} |j|||||||| } | d} |j| } |j | } d}|l|j | j }|j\}}t}|| j||z|j|j||z}| s| f| ddz}||f|zS|St|| | j| jS)r]Nr^rr$)rOrPrr )rr rrNrqr8r&r*rrrZrrr )rvrrrrrJrrrrrrTrrPrOr9r:rkrs rDrez#BloomForTokenClassification.forwards+>&1%rsR  LL$;;)>9. :, !;J   H %)Ju||)J)JEKK)J\a\h\h)JX5<<5<<uPTY^YeYe& Q%,, Q5<< Qu|| $ 5>>**  % %R.RYYR.jryy>;$+;$|"?""E%EEP z +_z  z z  p %9p  p fU "6U U pP 4P P f rF