i*?dZddlZddlZddlmZddlZddlmZddlmZm Z m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZm Z m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/e(r ddl0m1Z1ddl2m3Z3e*jhe5Z6GddejnZ8GddejnZ9GddejnZ:GddejnZ;GddejnZ<Gdd ejnZ=Gd!d"ejnZ>Gd#d$eZ?Gd%d&ejnZ@e'Gd'd(e#ZAGd)d*eAZBe'Gd+d,eAZCe'd-.Gd/d0eAeZDe'Gd1d2eAZEe'd3.Gd4d5eAZFe'Gd6d7eAZGe'Gd8d9eAZHgd:ZIy);zPyTorch UMT5 model.N)Union)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torch_flex_attn_availableis_torchdynamo_compilingloggingtorch_compilable_check)is_flash_attention_requested) UMT5Config) BlockMask)make_flex_block_causal_maskc&eZdZdfd ZdZxZS) UMT5LayerNormct|tjt j ||_||_y)ze Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s p/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/umt5/modeling_umt5.pyr)zUMT5LayerNorm.__init__?s1 ll5::k#:; #c|jtjjdj dd}|tj ||j zz}|jjtjtjfvr%|j|jj}|j|zS)NT)keepdim) tor+float32powmeanrsqrtr.r-dtypefloat16bfloat16)r/ hidden_statesvariances r3forwardzUMT5LayerNorm.forwardGs !##EMM266q9>>r4>P% Ht?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r4)gư>)__name__ __module__ __qualname__r)rC __classcell__r2s@r3r&r&>s $ +r4r&c*eZdZdeffd ZdZxZS)UMT5DenseActDenseconfigc^t|tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_ yNFbias)r(r)rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr/rKr2s r3r)zUMT5DenseActDense.__init__Ysn ))FNNFKKeD))FKKeDzz&"5"56 &--.r4c|j|}|j|}|j|}t|jj t jr|j|jj jk7r`|jj jt jk7r/|j|jj j}|j |}|SN) rSrYrW isinstancerTr-r+Tensorr>int8r9r/rAs r3rCzUMT5DenseActDense.forward`s . /  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r4rDrErFr"r)rCrGrHs@r3rJrJXs/z/ r4rJc*eZdZdeffd ZdZxZS)UMT5DenseGatedActDenserKct|tj|j|j d|_tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_yrM)r(r)rrPrQrRwi_0wi_1rTrUrVrWr rXrYrZs r3r)zUMT5DenseGatedActDense.__init__ps IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r4c,|j|j|}|j|}||z}|j|}t |j j tjr|j|j j jk7r`|j j jtjk7r/|j|j j j}|j |}|Sr\) rYrerfrWr]rTr-r+r^r>r_r9)r/rA hidden_gelu hidden_linears r3rCzUMT5DenseGatedActDense.forwardxshhtyy78  -0 #m3  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r4rarHs@r3rcrcos/z/r4rcc*eZdZdeffd ZdZxZS) UMT5LayerFFrKct||jrt||_nt ||_t |j|j|_ tj|j|_ y)Nr1)r(r) is_gated_actrcDenseReluDenserJr&rQlayer_norm_epsilon layer_normrrUrVrWrZs r3r)zUMT5LayerFF.__init__s_    "8"@D "3F";D 'F>+B+B*CD,,  4<<eD4<<eD4<<eD4>>4<FFq!QPQRr4cd}|j}|j}|jsC|dz}||dkDjtj |zz }t j |}n*t j|t j| }|dz}||k}t j|j|z tj||z z }|||z z}||jtj z}t j|t j||dz }|t j|||z }|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr6r!)rzr{rxr9r+longabsmin zeros_likelogfloatmath full_likewhere) r/relative_positionrelative_buckets num_buckets max_distance max_exactis_small log_ratiorelative_position_if_larges r3_relative_position_bucketz'UMT5Attention._relative_position_bucketsA*99 ;;  A K !2Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $y0II/557)CDtxxP\_hPhGii y!89 %.ejj1I%I"%*YY &8RT_bcTc(d& " EKK2CE_``r4c| |jjj}|.tj|tj |dddf}n |dddf}tj|tj |dddf}||z }|j |}|j|} | jgdjd} | S)z%Compute binned relative position biasN)r>device)r6rr!r) rr-rr+arangerrr unsqueeze) r/ query_length key_lengthrcache_positioncontext_positionmemory_positionrrelative_position_bucketvaluess r3 compute_biaszUMT5Attention.compute_biass >1188??F  !$||L SYZ[\^b[bc -ag6 ,,zFSTXZ[T[\+.>>#'#A#ABS#T --.FG *44Q7 r4rAencoder_hidden_statespast_key_valuesattention_maskrch|jdd\}}|du}|j|} | j|d|j|jj dd} d} |Qt |trA|jj|j} |r |j} n|j} n|} |r|n|} |rK|I| rG| j|jj} | j|jj}n|j!| } |j#| }| j|d|j|jj dd} |j|d|j|jj dd}|T|s|nd}| j%| ||jd|i\} }|r)t |trd|j|j<t'j(| | j dd}|||j+zn|}| jd}|j,s;t'j.d|j||f|j0|j2 }n1|j5|||j0| }|dddd| dddf}|#|ddddddd| jdf}||z}|}||z }t6j8j;|j=d j?|}t6j8jA||j@|jB }t'j(||}|j ddjE}|j||d}|jG|}||fS) Nr6r7r!FrTr)rr>)rrdim)ptraining)$shaperrrr} transposer]r is_updatedgetrvcross_attention_cacheself_attention_cachelayerskeysrrrupdater+matmulget_seq_lengthryzerosrr>rr functionalsoftmaxrtype_asrWr contiguousr)r/rArrrr batch_size seq_lengthis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_statesscoresreal_seq_lengthr position_bias causal_maskposition_bias_masked attn_weights attn_outputs r3rCzUMT5Attention.forwards"/!4!4Ra!8 J3$>vvm, #((RtG^G^_iijkmno   &:oGZ+[(3377GJ!'6'L'L$'6'K'K$#2 2D.- /"=*-44T^^DIIJ/66t~~FMML/J66.1L#RtG^G^_iijkmnoJ',,ZT\\4KbKbcmmnoqrsL*7It+?+F+F dnn?OQ_>`,( L&*_FY*ZAEO..t~~>lJ,@,@A,FGL[Kf*'E'E'GGlv%%b) //!KKDLL*j9&--W]WcWcM!--FMMR`.M*!Q a*?@M  %(Aq2HJ4D4DR4H2H)HIK)K7M,&&}},,V\\^,DLLVT }},,\T\\TXTaTa,b ll<> !++Aq1<<> !&&z:rB ff[) L((r4)FN)NNNNNN)rDrErF__doc__intr)r+r^rrrr rCrGrHs@r3ruruskSSWZk6%,, - ^ $6:(,.2.2 R)||R) %||d2R) R)  t+ R)  t+ R)r4ruc:eZdZddedzffd Z ddZxZS)UMT5LayerSelfAttentionNrvct|t|d||_t |j |j |_tj|j|_ y)NTryrvrm) r(r)ru SelfAttentionr&rQrprqrrUrVrWr/rKrvr2s r3r)zUMT5LayerSelfAttention.__init__YsN *6t_hi'Fr+r?finforrisinfanyclamprx)r/rArrencoder_attention_maskr use_cacheoutput_attentionsrself_attn_weights max_dtype clamp_valuecross_attn_weightsdo_cross_attentionrs r3rCzUMT5Block.forwards,94::a= )+) , ( (   %-- / M$7$78< >Gr4r\)NNNNFFNrrHs@r3rrs//#*/"#7r4rc`eZdZdZdeffd ZdejdejfdZxZ S)UMT5ClassificationHeadz-Head for sentence-level classification tasks.rKc(t|tj|j|j|_tj |j|_tj|j|j|_ y)N)r) r(r)rrPrQdenserUclassifier_dropoutrW num_labelsout_projrZs r3r)zUMT5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r4rArc|j|}|j|}tj|}|j|}|j |}|Sr\)rWrr+tanhr r`s r3rCzUMT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r4) rDrErFrr"r)r+r^rCrGrHs@r3rrs/7EzE U\\ellr4rcteZdZUeed<dZdZdZdgZdgZ e dZ e jdZdZy ) UMT5PreTrainedModelrK transformerTrrTcvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r+tensorrr)r/r input_mask dummy_inputss r3rz UMT5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 r4cr|jj}t|tr$t j |j |dzy t|ttttfrt j|jj d|dzt|drE|jjs/t j|jj d|dzt|drpt j|j j d||jj"dzzt j$|j j&y y t|t(rft|drYt j|j*j d|dzt j$|j*j&y y t|t,r9t j|j.j d||jj"dzzt|j.dr?|j.j&)t j$|j.j&t j|j0j d||jj"dzzt|j0drA|j0j&*t j$|j0j&y y y t|t2r9t j|j4j d||jj"dzzt|j4dr?|j4j&)t j$|j4j&t j|j6j d||jj8dzzt|j6drA|j6j&*t j$|j6j&y y y t|t:rt j|j<j d||jj"dzzt|j<dr?|j<j&)t j$|j<j&t j|j>j d||jj"dzzt|j>dr?|j>j&)t j$|j>j&t j|j6j d||jj8dzzt|j6drA|j6j&*t j$|j6j&y y y t|t@rP|jj"}|jjB}|jjD}t j|jFj d|||zdzzt j|jHj d||dzzt j|jJj d||dzzt j|jLj d|||zdzz|jNr3t j|jPj d||dzzy y y ) zInitialize the weights?)r<stdlm_head qa_outputs classifierrON))rKinitializer_factorr]r&init constant_r- UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrrrQzeros_rOUMT5ForTokenClassificationrrrr rJrSrTrRrcrerfrur|r~rrrrryr)r/modulefactorrQr}rs r3 _init_weightsz!UMT5PreTrainedModel._init_weightss>// fm , NN6==&3, 7  , (    LL--CVc\ Jvy)$++2Q2Q V^^22&3,Ov|, V..55CVPTP[P[PcPchlOlEmn F--223- : ;v|, V..55CVc\R F--223- 6 7 LL,,3Ft{{GZGZ_cFcs!%!C!C{{// ! )6  &// @%.sCRCx%8%>%>%@#qr'"$:&!  PQ Q&&'8D'@,O  r4N)rDrErFr"__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr+no_gradr.r8r4r3rrsb %&*#!$ !F U]]_@p@pD!r4rc eZdZfdZdZ ddZ ddeejdfdejdejde d e f d Z e dejd e d e d ejdejde f dZxZS) UMT5Stackc t||tj|j|j |_|j|_tjt|jDcgc]}t||c}|_ t|j |j|_tj |j"|_d|_|j)ycc}w)NrrmF)r(r)rr vocab_sizerQ embed_tokensrxrrange num_layersrblockr&rpfinal_layer_normrUrVrWgradient_checkpointing post_init)r/rKir2s r3r)zUMT5Stack.__init__Us  LL):):FNNK ++]]ERXRcRcLd#eqIf$B#ef -fnn&B[B[ \zz&"5"56 ',#  $fs7C9c||_yr\)rFr/new_embeddingss r3set_input_embeddingszUMT5Stack.set_input_embeddingsas *r4c d||n|jj}||n|jj}| | n|jj} | | n|jj} |$|"|j rdnd} t d| d| d|&|j}|jd|d}n8||jdd}n"|j rdnd} t d| d| d |jr%|jr|rtjd d }|(|j t d |j|}|\}}|d ur|j st d|d|j rf|rr|p|jjr5tt!|jt!|j}n%t!|j}n|j sd}||j#nd}| %t%j&|||z|j(} |1t+s'||z}t%j,|||j(}|j r2|j/||| t1|tr |j2n||}n]|Y|ddddddf}|j5|j6}d|z t%j8|j6j:z}nd}|j rO|M|j\}}}||f}|!t%j,||j(}|j=|}nd}| rdnd}|rdnd}|r|j rdnd}|j?|}tA|jBD]D\}}| r||fz}||||||||| }|d}|s&||dfz }|j s<||dfz }F|jE|}|j?|}| r||fz}| stGd|||||fDStI|||||S)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer7zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzrrA)rrrrrr!r6c3$K|]}|| ywr\rA).0rs r3 z$UMT5Stack.forward..s = s)last_hidden_staterrA attentionscross_attentions)%rKrroutput_hidden_statesuse_return_dictrxr3rrrKrrrrFis_encoder_decoderr r rr+rrrr,_update_causal_maskr]rr9r>rrinvert_attention_maskrW enumeraterIrJtupler) r/rrrrrUrrrr] return_dictrkwargserr_msg_prefix input_shaperrpast_key_values_lengthmask_seq_lengthrencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrArM layer_module layer_outputss r3rCzUMT5Stack.forwardds"+!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!, J  ?? #LTFRg!hii ??_4;;11&9$DKK8,dkk:Z'O'3$++&FO#OETE`!?!?!Afg  !"\\&(>(KTaThThN  !*B*D4zAO"ZZ OML`L`aN ??22o/BC 44$!K '(D$)9:K%..}/B/B.CK, Mrr)cudaxpunpu)r rKr_attn_implementationr]r+r^r$ris_compileabler_ignore_causal_mask_sdparr>rget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrtyperr_unmask_unattended) r/rrtrrrpast_seen_tokensusing_compilable_cacher>ryrzr min_dtypes r3r`zUMT5Stack._update_causal_masks ( 4)~/D.I.I.K%% ;; + +/? ?.%,,7!<^!L! ! @O?Z?99;`aCRC^!?!?di ;; + +v 5>T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCKQZ[Kr4ryrzr>rc ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer>rr!)diagonalrVr7r)rr+rrfullrtriurreshapeexpandr5rr9 masked_fill) rryrzr>rrrerr mask_length padding_masks r3rz?UMT5Stack._prepare_4d_causal_attention_mask_with_cache_positionKs>  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r4) NNNNNNNNNNN)F)rDrErFr)rQrCrr+r^r boolr` staticmethodrr>rrGrHs@r3rCrCTs + "#!` R#( BellK78BllB B  B  BH4 444{{ 4  4  44r4rCceZdZUdZdZeed<dddZfdZdZ dZ e dd e jd zd e jd zd e jd zd e jd zdeee jd zded zde j$d zde j$d zded zded zded zded zde jd zdee jezfdZxZS)r"ao Examples: ```python >>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5rK shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightct||tj|j|j |_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_|j!yNFT)r(r)rrrErQr'copydeepcopyrxrrCencodernum_decoder_layersrHdecoderrLr/rKencoder_configdecoder_configr2s r3r)zUMT5Model.__init__s  ll6#4#4fnnE v.$)!#(  0 v.$(!$*$=$=! 0  r4c|jSr\r'r/s r3get_input_embeddingszUMT5Model.get_input_embeddings {{r4c~||_|jj||jj|yr\r'rrQrrOs r3rQzUMT5Model.set_input_embeddings-$  )).9 )).9r4Nrrrrencoder_outputsrrUdecoder_inputs_embedsrrr]rdrrc F| | n|jj} | | n|jj} ||j|||| | | }nI| rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}|j||||||| | | | |  }| s||zSt|j|j|j|j|j|j|j|jS) ah input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```NrrrUrr]rdrr!r6rZrAr[ rrrUrrrrrr]rdr)rZrdecoder_hidden_statesdecoder_attentionsr\encoder_last_hidden_staterencoder_attentions)rKrr^rr]rlenrrrZrrAr[r\)r/rrrrrrrUrrrr]rdrrerAdecoder_outputss r3rCzUMT5Model.forwardsRF"+!6IDKKBF(,-159!%)-,0#'26s ##d*s ))D0s !++d2 s !& 0 04 7 s uU%6%6784? s s ||d*s  %||d2s $;s  $;s #Tks D[s ((4/s u  !$6 6!s s r4r"z< UMT5 Model with a `language modeling` head on top. ) custom_introc!eZdZdZdZddddZfdZdZdZe dd e jdzd e jdzd e jdzd e jdzd eee jdzdedzde jdzde jdzde jdzdedzdedzdedzdedzde jdzdee jezfdZde jfdZxZS)r#a Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```rr)rrzlm_head.weightct|||j|_t j |j |j|_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_t j"|j|j d|_|j'y)NFTrN)r(r)rQ model_dimrrrEr'rrrxrrCrrrHrrPrrLrs r3r)z%UMT5ForConditionalGeneration.__init__Ms  ll6#4#4fnnE v.$)!#(  0 v.$(!$*$=$=! 0 yy1B1BO  r4c|jSr\rrs r3rz1UMT5ForConditionalGeneration.get_input_embeddingscrr4c~||_|jj||jj|yr\rrOs r3rQz1UMT5ForConditionalGeneration.set_input_embeddingsgrr4NrrrrrrrUrlabelsrrr]rdrrc | | n|jj} | | n|jj} ||j|||| | | }nI| rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}| |||j| }|j||||||| | | | | }|d}|jjr||jdzz}|j|}d}| ^td }| j|j} ||jd |j!d | jd }| s|f|ddz|z}||f|zS|St#|||j$|j&|j(|j*|j,|j&|j( S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nrrr!r6rrrr0 ignore_indexr7 losslogitsrrrr\rrr)rKrr^rr]rrr8rr)rrrr9rrrrrrAr[r\rZ)r/rrrrrrrUrrrrr]rdrrerArsequence_output lm_logitsrloss_fctoutputs r3rCz$UMT5ForConditionalGeneration.forwardls"N"+!6IDKK=A(,26:>*.!%)-,0#'26N ##d*N ))D0N !++d2 N !& 0 04 7 N uU\\23d: N N ((4/N  %0047N   4'N $;N  $;N #TkN D[N ((4/N " u  !O 3#N N b)ELL)r4r#ceZdZdZdZddiZfdZdZdZe dd e jdzd e jdzd e jdzd e dzd e dzde dzdee jezfdZxZS)r$a Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rrrct||tj|j|j |_tj|}d|_ d|_ t||_ |jy)NF)r(r)rrrErQr'rrrr_rCrrL)r/rKrr2s r3r)zUMT5EncoderModel.__init__sb  ll6#4#4fnnE v.#( ,1) 0  r4c|jSr\rrs r3rz%UMT5EncoderModel.get_input_embeddings%rr4cH||_|jj|yr\)r'rrQrOs r3rQz%UMT5EncoderModel.set_input_embeddings)s$  )).9r4NrrrUrr]rdrc h||n|jj}|j||||||}|S)aQ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r)rKr^r) r/rrrUrr]rdrers r3rCzUMT5EncoderModel.forward-sHF&1%|j@|jB|jD|jF|jH S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrrrUrrrr]rdrr!z7All examples must have the same number of tokens.r7 regressionsingle_label_classificationmulti_label_classificationr)%rKr^NotImplementedErrorr2rDr3r8req eos_token_idr9rrr+unique_consecutivesumnumelrrr problem_typer r>rrrsqueezerrrrrrr\rrr)r/rrrrrrUrrrrr]rdrerreos_maskrrlr0sentence_representationrrrrs r3rCz%UMT5ForSequenceClassification.forwardpsN`&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9+'"7/!5##  "!*<< 8 89<<_=S=ST  $ $X\\!_ 5 ; ; = B E &5%:%:" A{"1(A+">"C"CJPRT_"`abdfhiai"j))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r4) NNNNNNNNNNNN)rDrErF"_keys_to_ignore_on_load_unexpectedr"r)rr+rr^listrrrcrrCrGrHs@r3rr^sU+s)s&z.2.259:>:>26:>*.!%)-,0#'A ##d*A  t+A !++d2 A !& 0 04 7 A e//047 A ((4/A  %0047A   4'A $;A  $;A #TkA D[A  0 0A A r4rceZdZdgZdeffd Ze ddejdzdejdzdejdzdejdzd e dzd e dzd e dzd e eje zfd Z xZ S)r+rrKc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr\) r(r)r r$rrrUrrWrPr0rrLrZs r3r)z#UMT5ForTokenClassification.__init__sj   +++F3zz&";";< ))F$6$68I8IJ r4NrrrUrrr]rdrc ||n|jj}|j||||||} | d} |j| } |j | } d} |)F  U !% 1 1) < !*!6IDKK=A371526:>!%)-,0#'I ##d*I ))D0I !++d2 I !& 0 04 7 I uU\\23d: I ))D0I ''$.I ((4/I  %0047I $;I  $;I #TkI D[I u  !$G G!I I r4r%)r$r#r%rr+r"r)Jrrrtypingrr+rtorch.nnrrrrTr r  activationsr cache_utilsr r r generationrmodeling_attn_mask_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrrr utils.genericr configuration_umt5r"!torch.nn.attention.flex_attentionr#integrations.flex_attentionr$ get_loggerrDrModuler&rJrcrkrurrrrrrCr"r#r$rr+r%__all__rAr4r3rs  AA&!CC)>9.:* !;J   H %+BII+4 .RYY<"))$v)BIIv)rRYY4bii8B*BLRYY$j!/j!j!Zm#m` h #h h V J)#6J)  J)ZX*XXv N $7N  N bG !4G G Tp 2p p f r4