iHddlZddlmZmZddlZddlmZddlmZmZm Z ddl m Z ddl mZddlmZmZdd lmZmZdd lmZmZdd lmZdd lmZmZmZmZmZm Z dd l!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3m4Z4e,jje6Z7GddeZ8GddejrZ:GddejrZ;Gdde3Z<edd9dZ=ee=Gdd ejrZ>Gd!d"eZ?e+Gd#d$e&Z@e+Gd%d&e@ZAGd'd(ejrZBe+d)*Gd+d,e@ZCe+d-*Gd.d/e@ZDe+d0*Gd1d2e@ZEe+Gd3d4e@ZFe+d5*Gd6d7e@ZGgd8ZHy):N)LiteralOptional)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)PreTrainedConfiglayer_type_validation)use_kernel_func_from_hubuse_kernelized_func)create_bidirectional_mask(create_bidirectional_sliding_window_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ROPE_INIT_FUNCTIONSRopeParameters)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuplecheck_model_inputs)eager_attention_forward)Gemma3RotaryEmbedding rotate_halfcF8eZdZdZdZdgZdddZfdZ d2d edzd edzd edzd edzd edzde dzdedzde dzde dzde dzde dzdedzdedzdedzdedzdedzde dzde dzde e dzde edefdzdedzde dzd e dzd!e dzd"e dzd#ed$d%e dzd&e dzd'e dzd(e dzd)e dzd*edzd+e dzd,e dzfDfd- Zd3d.Zfd/Zed0Zej*d1ZxZS)4ModernBertConfigaO This is the configuration class to store the configuration of a [`ModernBertModel`]. It is used to instantiate an ModernBert model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ModernBERT-base. e.g. [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 50368): Vocabulary size of the ModernBert model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`ModernBertModel`] hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 1152): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 22): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu"` if not specified. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_cutoff_factor (`float`, *optional*, defaults to 2.0): The cutoff factor for the truncated_normal_initializer for initializing all weight matrices. norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. norm_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the normalization layers. pad_token_id (`int`, *optional*, defaults to 50283): Padding token id. eos_token_id (`int`, *optional*, defaults to 50282): End of stream token id. bos_token_id (`int`, *optional*, defaults to 50281): Beginning of stream token id. cls_token_id (`int`, *optional*, defaults to 50281): Classification token id. sep_token_id (`int`, *optional*, defaults to 50282): Separation token id. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. layer_types (`list`, *optional*): Attention pattern for each layer. rope_parameters (`dict`, *optional*): Dictionary mapping attention patterns (`"full_attention"`, `"sliding_attention"`) to `RopeParameters`. Each value should be a dictionary containing `rope_type` and optional scaling parameters. local_attention (`int`, *optional*, defaults to 128): The window size for local attention. embedding_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the embeddings. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layers. mlp_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the MLP layers. decoder_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the decoder layers. classifier_pooling (`str`, *optional*, defaults to `"cls"`): The pooling method for the classifier. Should be either `"cls"` or `"mean"`. In local attention layers, the CLS token doesn't attend to all tokens on long sequences. classifier_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the classifier. classifier_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the classifier. classifier_activation (`str`, *optional*, defaults to `"gelu"`): The activation function for the classifier. deterministic_flash_attn (`bool`, *optional*, defaults to `False`): Whether to use deterministic flash attention. If `False`, inference will be faster but not deterministic. sparse_prediction (`bool`, *optional*, defaults to `False`): Whether to use sparse prediction for the masked language model instead of returning the full dense logits. sparse_pred_ignore_index (`int`, *optional*, defaults to -100): The index to ignore for the sparse prediction. reference_compile (`bool`, *optional*): Whether to compile the layers of the model which were compiled during pretraining. If `None`, then parts of the model will be compiled if 1) `triton` is installed, 2) the model is not on MPS, 3) the model is not shared between devices, and 4) the model is not resized after initialization. If `True`, then the model may be faster in some scenarios. This argument is deprecated and will be removed in a future version. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings Examples: ```python >>> from transformers import ModernBertModel, ModernBertConfig >>> # Initializing a ModernBert style configuration >>> configuration = ModernBertConfig() >>> # Initializing a model from the modernbert-base style configuration >>> model = ModernBertModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` modernbertpast_key_valuesgAg@)globallocalcb|dk(r|tjdd}t| ||y)Nreference_compilezThe `reference_compile` argument is deprecated and will be removed in `transformers v5.2.0`Use `torch.compile()` directly on the model instead.)logger warning_oncesuper __setattr__)selfnamevalue __class__s {/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/modernbert/modular_modernbert.pyr2zModernBertConfig.__setattr__s; & &5+<   G E D%(N vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headshidden_activationmax_position_embeddingsinitializer_rangeinitializer_cutoff_factornorm_eps norm_bias pad_token_id eos_token_id bos_token_id cls_token_id sep_token_idattention_biasattention_dropout layer_typesrope_parametersfull_attentionsliding_attentionlocal_attentionembedding_dropoutmlp_bias mlp_dropout decoder_biasclassifier_poolingclsmeanclassifier_dropoutclassifier_biasclassifier_activationdeterministic_flash_attnsparse_predictionsparse_pred_ignore_indexr.tie_word_embeddingsc# b| |_||_| |_||_||_|"|_||_||_||_||_ ||_ ||_ ||_ | |_ | |_| |_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |_|!|_|j0dvrtAd|j0d||_!|#jEdd|_#|jBBtI|jD$cgc]}$tK|$|jFzrdnd c}$|_!tM|jB|j||_'tQ%|di|#ycc}$w) NrVzQInvalid value for `classifier_pooling`, should be either "cls" or "mean", but is .global_attn_every_n_layersr rOrN)*rDrFrErGrHr_r9r?r:r;r<r=r@rArBrCrIrJr>rPrQrRrSrTrUrYrZr[r\r]r^r. ValueErrorrKgetrbrangeboolr rLr1__init__)&r3r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrPrQrRrSrTrUrYrZr[r\r]r^r.r_kwargsir6s& r7rhzModernBertConfig.__init__sL)((((#6 $'>$&!2!2#6 !2)B&  ",!2!2.!2  &("4"4.%:"(@%!2(@%!2  " "/ 9cdhd{d{c||}~ '+1**5QST*U'    #t556 (,A0O0O,O'P#Vff D  d..0F0FG. "6" s #F,c |jdd}ddiddid}|j |jn||_|<|jdj||jdj||jjdddi|jd<|jdj d|jd|j d |jjdddi|jd<|jdj d|jd |j d |j |j| |S) N rope_scaling rope_typedefault)rOrNrNrO rope_thetaglobal_rope_thetar+local_rope_thetar,) ignore_keys)poprLupdatere setdefault default_thetastandardize_rope_params validate_rope)r3ignore_keys_at_rope_validationrirldefault_rope_paramss r7convert_rope_params_to_dictz,ModernBertConfig.convert_rope_params_to_dictsuzz.$7 #.y!9*I6 8<7K7K7Wt33]p  #  !1 2 9 9, G  !4 5 < <\ J    # #$4 5 =6A95MD !1 2 -.99 &**%8$:L:LX:VW     # #$7 8 @9Di8PD !4 5 01<< &**%79K9KG9TU $$& 'EF r8cHt|}|jdd|S)Nr.)r1to_dictrs)r3outputr6s r7r}zModernBertConfig.to_dicts#" &- r8c |jdzS)zKHalf-window size: `local_attention` is the total window, so we divide by 2.r#rPr3s r7sliding_windowzModernBertConfig.sliding_window#s##q((r8c|dz|_y)zFikjirrFNNrFrTrWrFrFFiNTN)__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencervr2intstrfloatrglistdictrrrhr{r}propertyrsetter __classcell__r6s@r7r(r(1sc JJ#4"5(8#'+?Y#@ $;AY#B#&*CY#D $;EY#F"D[GY#v> ))))r8r(ceZdZdZdeffd Z d dejdzdejdzdejfdZ xZ S) ModernBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. configcdt|||_tj|j |j |j|_tj|j |j|j|_ tj|j|_y)N) padding_idxepsbias)r1rhrr Embeddingr9r:rDtok_embeddings LayerNormrBrCnormDropoutrQdropr3rr6s r7rhzModernBertEmbeddings.__init__3sw   ll6+<+P>P^d^q^qrLL!3!3vO_O_` JJv778 r8N input_ids inputs_embedsreturnc|"|j|j|}|S|j|j|j|}|Sr)rrr)r3rr hidden_statess r7forwardzModernBertEmbeddings.forward:sS  $ IIdii &>?M!IIdii0C0CI0N&OPMr8NN) rrrrr(rhtorch LongTensorTensorrrrs@r7rr.sR9/9_c))D0HM W[H[ r8rc`eZdZdZdeffd ZdejdejfdZxZ S) ModernBertMLPa6Applies the GLU at the end of each ModernBERT layer. Compared to the default BERT architecture, this block replaces :class:`~transformers.model.bert.modeling_bert.BertIntermediate` and :class:`~transformers.model.bert.modeling_bert.SelfOutput` with a single module that has similar functionality. rct|||_tj|j t |jdz|j|_ t|j|_ tj|j|_tj|j|j |j|_y)Nr#r)r1rhrrLinearr:rr;rRWir r>actrrSrWors r7rhzModernBertMLP.__init__Ks  ))F..F4L4L0MPQ0QX^XgXgh&223JJv112 ))F44f6H6Hv_r8rrc|j|jdd\}}|j|j|j ||zS)Nr#dim)rchunkrrr)r3rinputgates r7rzModernBertMLP.forwardSsIggm,221"2= twwtyy%4!7899r8) rrrrr(rhrrrrrs@r7rrDs2 `/`:U\\:ell:r8rc|eZdZd deffd Ze d dedzdeddedzdedzde d e ff fd Z xZ S) ModernBertRotaryEmbeddingNrc&t|||yr)r1rh)r3rdevicer6s r7rhz"ModernBertRotaryEmbedding.__init__Ys (r8rz torch.deviceseq_len layer_typerz torch.Tensorc(t|||||Sr)r1compute_default_rope_parameters)rrrrr6s r7rz9ModernBertRotaryEmbedding.compute_default_rope_parameters\sw6vvwPZ[[r8rNNNN) rrrr(rh staticmethodrrrtuplerrrrs@r7rrXs)/)*.+/"!% \ 4'\(\t\$J \ ~u$ % \\r8rrotary_pos_embcb|j}|j|}|j|}|j|zt|j|zz}|j|zt|j|zz}|j ||j |fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )dtype unsqueezerr&to)qkcossin unsqueeze_dimoriginal_dtypeq_embedk_embeds r7apply_rotary_pos_embrfs&WWN -- &C -- &Cwwy3;qwwy#9C#?@Gwwy3;qwwy#9C#?@G ::n %wzz.'A AAr8ceZdZdZd dededzffd Z d dejde ejejfdzdejdzd e e d e ejejdzff d Z xZ S)ModernBertAttentionaPerforms multi-headed self attention on a batch of unpadded sequences. If Flash Attention 2 is installed, this module uses Flash Attention to improve throughput. If Flash Attention 2 is not installed, the implementation will use PyTorch's SDPA kernel, which requires padding and unpadding inputs, adding some overhead. See `forward` method for additional details. Nr layer_idxcRt|||_||_|j|j zdk7r&t d|jd|j d|j|_|j|_|j|j z|_ tj|jd|jz|j z|j|_ |j|dk(r|jdz|_nd|_d |_tj|j|j|j|_|jd kDr%tj$|j|_ytj&|_y) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r rrOFr)r1rhrrr:r=rdrJr\head_dimrrrIWqkvrKr is_causalrrIdentityout_dropr3rrr6s r7rhzModernBertAttention.__init__sz  "    : : :a ?#F$6$6#77mntoIoInJJKL "(!9!9(.(G(G%**f.H.HH II   DMM 1F4N4N NU[UjUj     i (,? ?#)"7"7!";D "&D ))F..0B0BI^I^_@F@X@X[^@^ 6#;#;< dfdododq r8rposition_embeddingsattention_maskrirc |jdd}|j|}|jg|dd|j}|j d\}}} |j dd}|j dd}| j dd} |\} } t ||| | d\}}t} |jjdk7rt|jj} | |||| |f|jr |jnd |jd z|j|jd |\} }| jg|dj!} |j#|j%| } | |fS) Nrr rrr#)reagerr)dropoutscalingr deterministic)shaperviewrunbind transposerr$r_attn_implementationrtrainingrJrr\reshape contiguousrr)r3rrrri input_shapeqkv query_states key_states value_statesrrattention_interface attn_output attn_weightss r7rzModernBertAttention.forwards$))#2. ii &chh: :Q::DMM:141C. j,#--a3 ))!Q/ #--a3 &S#7 jRUWZjk#l j5 ;; + +w 6"9$++:Z:Z"[ $7      % /3mmD**MM4'..77 %  % ! \*k));;;;FFH mmDGGK$89 L((r8rr)rrrrr(rrhrrrrrrrrs@r7rrsr/rC$Jr@IM.2 ')||')#5<<#=>E') t+ ') +, ') u||U\\D00 1 ')r8rc eZdZd dededzffd Z d dejdejdzdejdzdee d ejf d Z xZ S) ModernBertEncoderLayerNrrct|||_||_|dk(rt j |_n;t j|j|j|j|_t|||_ t j|j|j|j|_ t||_|j ||_y)Nrr)rr)r1rhrrrr attn_normrr:rBrCrattnmlp_normrmlprKattention_typers r7rhzModernBertEncoderLayer.__init__s  " >[[]DN\\&*<*<&//X^XhXhiDN'vK  V%7%7V__SYScScd  ($00;r8rrrrirc |j|j|f||d|\}}||z}||j|j|z}|S)N)rr)rrr r)r3rrrrir_s r7rzModernBertEncoderLayer.forwardsg# NN= ) 3)    Q & 3 %}1M(NN r8rr) rrrr(rrhrrrrrrrs@r7rrsx </  t9j@|j>yyt|tBr|jDD]}|jF}|jH|d k7rtJ|jH|}||j| \}}t9jLtO||d|t9jLtO||d|yy)Nr rstdctj|jd| |z|zt|tj r-|j tj|j yyy)Nr)rXrab)init trunc_normal_weight isinstancerrrzeros_)rr cutoff_factors r7 init_weightz.init_weight sd     .3&#%  &")),;;*KK ,+-r8rr)inout embedding final_outr rrr!rn)r _inv_freq_original_inv_freq)(rrArModulerr@mathsqrtr<r:rrrrrrrrModernBertPredictionHeaddenseModernBertForMaskedLMdecoder#ModernBertForSequenceClassificationModernBertForMultipleChoice ModernBertForTokenClassificationModernBertForQuestionAnswering classifierrrones_rrrrrKrrmrcopy_getattr) r3rrstdsr rope_init_fn curr_inv_freqr rs @r7 _init_weightsz'ModernBertPreTrainedModel._init_weightssd ==  M - - -++//;;00499S4;;C`C`=`3aa6600$6   f2 3 --tK/@ A  .  4: .  4; /  3 4  T$Z 0  4; /  8 9  d5k 2  5 6 U 4  3+0.    ))4 +< =  - JJv}} %{{& FKK('  9 :$00 ^ %EE ##J/9<#6v7G7G 7S#TL#/ *#U q 76j\+CDmT 76j\9K+LM}]  ^;r8attn_implementationN is_init_checkrc |dn|}t|||S#ttf$rt|||cYSwxYw)zR Checks and dispatches to hhe requested attention implementation. flash_attention_2)r7r8)r1%_check_and_adjust_attn_implementationrd ImportError)r3r7r8requested_attn_implementationr6s r7r;z?ModernBertPreTrainedModel._check_and_adjust_attn_implementation@sh CVC^,?dw )7@$AQ^A K( 7@$7}A  s  ==)F)rrrr(__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsrno_gradrr$r6rrgr;rrs@r7rrs &*#/1IJN"&0) U]]_:^BII:^:^zFK#&:>B r8rceZdZdeffd ZdZdZee d de jdzde jdzde jdzd e jdzd e e d ef d ZxZS)ModernBertModelrc t||||_t||_t j t|jDcgc]}t||c}|_ t j|j|j|j|_t!||_d|_|j'ycc}w)Nr)rF)r1rhrr embeddingsr ModuleListrfr<rlayersrr:rBrC final_normr rotary_embgradient_checkpointing post_initrs r7rhzModernBertModel.__init__Us   .v6mmHMfNfNfHg h9 #FI 6 h  ,,v'9'9vU[UeUef36B&+#  isCc.|jjSrrKrrs r7get_input_embeddingsz$ModernBertModel.get_input_embeddingsas---r8c&||j_yrrSrs r7set_input_embeddingsz$ModernBertModel.set_input_embeddingsds).&r8Nrr position_idsrrirc |du|duz r td||jdn|jd}| |jn |j}|&tj||j d}|j ||}t|x} ts'|j||d} td i| td i| d} i} |jjD]} |j||| | | <|jD](} | |f| | j| | jd|}*|j!|}t#| S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)rr)r input_embedsrrM)rr)last_hidden_staterc)rdrrrarangerrKrrrrrrKrOrMr rNr)r3rrrWrrirrrattention_mask_mapping mask_kwargsrr encoder_layers r7rzModernBertModel.forwardgsw  -t"; <YZ Z,9,E-%%a(9??[\K]%.%:!! @T@T   <<?II!LL)=Y NB0DI++ -"0K #<"Jk"J%M%\P[%\& " !++11 gJ.2oom\[e.f  + g"[[ M)5m6R6RS$7 8T8T$U M  6 ??r8r)rrrr(rhrTrVr"rrrrrrrrrrs@r7rIrISs / ./.2.204-1 ,@##d*,@ t+,@&&- ,@ ||d* ,@ +, ,@ ,@,@r8rIc\eZdZdeffd Zdej dej fdZxZS)r'rcJt|||_tj|j |j |j |_t|j|_ tj|j |j|j|_y)Nr)r1rhrrrr:rZr(r r[rrrBrCrrs r7rhz!ModernBertPredictionHead.__init__sq  YYv1163E3EvG]G]^ &667LL!3!3vO_O_` r8rrc`|j|j|j|Sr)rrr()r3rs r7rz ModernBertPredictionHead.forwards#yy$**]";<==r8) rrrr(rhrrrrrs@r7r'r's-a/a>U\\>ell>r8r'zd The ModernBert Model with a decoder head on top that is used for masked language modeling. ) custom_introc>eZdZddiZdeffd ZdZdejfdZ e e dd e jdzd e jdzd e jdzd e jdzd e jdzdeedee jezfdZxZS)r)zdecoder.weightz&model.embeddings.tok_embeddings.weightrctt||||_t||_t ||_tj|j|j|j|_ |jj|_ |jj|_|jy)Nr)r1rhrrIrr'headrrr:r9rTr*r]r^rQrs r7rhzModernBertForMaskedLM.__init__s   $V, ,V4 yy!3!3V5F5FVM`M`a !%!>!>(, (L(L% r8c|jSrr*rs r7get_output_embeddingsz+ModernBertForMaskedLM.get_output_embeddingss ||r8new_embeddingsc||_yrrh)r3rjs r7set_output_embeddingsz+ModernBertForMaskedLM.set_output_embeddingss % r8NrrrWrlabelsrirc |jd||||d|}|d}|jrK|I|jd}|j|jdd}||jk7} || }|| }|j |j |} d} |*|j| |fd|jji|} t| | |j|jS)NrrrWrrrr9losslogitsrrrc) rr]rrr^r*rf loss_functionrr9rrr) r3rrrWrrmrioutputsr[ mask_tokensrrrqs r7rzModernBertForMaskedLM.forwards $** )%'    $AJ  ! !f&8[[_F 1 6 6v||A K !D$A$AAK 1+ > K(Fdii(9:;  %4%%ffbAWAWb[abD!//))   r8NNNNN)rrr_tied_weights_keysr(rhrirrrlr!rrrrrrrrrrrs@r7r)r)s +,TU / &BII&.2.2,0-1&* ' ##d*'  t+' llT) ' ||d* '  t# ' +,'  u|| ~ -' ' r8r)z` The ModernBert Model with a sequence classification head on top that performs pooling. ceZdZdeffd Zee d dejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) r+rcnt|||j|_||_t ||_t ||_tjj|j|_ tj|j|j|_|j!yr)r1rh num_labelsrrIrr'rfrrrrYrrr:r/rQrs r7rhz,ModernBertForSequenceClassification.__init__s   ++ $V, ,V4 HH$$V%>%>? ))F$6$68I8IJ r8NrrrWrrmrirc d|jd||||d|}|d}|jjdk(r |dddf}n|jjdk(rw|=tj|j dd|j tj}||jdzjd |jd d z }|j|} |j| } |j| } d} ||jj|jd k(rd |j_ nl|jd kDrL|jtj k(s|jtj"k(rd|j_ nd|j_ |jjd k(rIt%} |jd k(r&| | j'|j'} n| | |} n|jjdk(r=t)} | | j+d|j|j+d} n,|jjdk(rt-} | | |} t/| | |j0|j2S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rorrWNrXr#)rrrrrTrkeepdim regressionsingle_label_classificationmulti_label_classificationrprc)rrrUronesrrrgrsumrfrr/ problem_typerzrlongrrsqueezerrrrrr) r3rrrWrrmrirtr[ pooled_outputrrrqloss_fcts r7rz+ModernBertForSequenceClassification.forwardse"$** )%'    $AJ ;; ) )U 2 1!Q$ 7  [[ + +v 5%!&%++BQ/8I8P8PX]XbXb""3^5M5Mb5Q!Q V V[\ V ]`n`r`rtasa!  "34  -0 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r8rv)rrrr(rhr!rrrrrrrrrrrs@r7r+r+s / .2.2,0-1&* C ##d*C  t+C llT) C ||d* C  t# C +,C  u|| 7 7C C r8r+zv The ModernBert Model with a token classification head on top, e.g. for Named Entity Recognition (NER) tasks. ceZdZdeffd Zee d dejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) r-rc`t|||j|_t||_t ||_tjj|j|_ tj|j|j|_|jyrr1rhrzrIrr'rfrrrrYrrr:r/rQrs r7rhz)ModernBertForTokenClassification.__init__Ls{   ++$V, ,V4 HH$$V%>%>? ))F$6$68I8IJ r8NrrrWrrmrirc f|jd||||d|}|d}|j|}|j|}|j|} d} |%>? ))F$6$68I8IJ r8NrrrWstart_positions end_positionsrirc |j|f||d|}|d}|j|}|j|}|j|} | j dd\} } | j dj } | j dj } d} |||j| | ||fi|} t| | | |j|jS)N)rrWrrrr)rq start_logits end_logitsrr) rrfrr/splitrrrsrrr) r3rrrWrrrirtr[rrrrrqs r7rz&ModernBertForQuestionAnswering.forwards$**  )%    $AJ II&78 II&78!23#)<<r<#: j#++B/::< ''+668   &=+D%4%%lJQ^ibhiD+%!!//))   r8rv)rrrr(rhr!rrrrrrrrrrs@r7r.r.s / *..2,0/3-1 # <<$&#  t+# llT) # , # ||d* # +,#  u|| ; ;# # r8r.z The ModernBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZdeffd Zee d dejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) r,rc8t||||_t||_t ||_tjj|j|_ tj|jd|_|jy)Nr)r1rhrrIrr'rfrrrrYrrr:r/rQrs r7rhz$ModernBertForMultipleChoice.__init__so   $V, ,V4 HH$$V%>%>? ))F$6$6: r8NrrrWrrmrirc ||jdn|jd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|jd||||d|}|d} |jj dk(rt j| jd| j} |,|jd j| j} n0t jdt j| j } | | | f} nS|jj d k(r:|jdd } | |jdzjd | z } |j| } |j!| } |j#| }|jd|}d}|t%j&}|||}t)|||j*|j,S)a& labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. NrrrorrWrYr)rrrXTr|rprc)rrsizerrrUrr\rargmaxrtensorrrrrfrr/rrrrr)r3rrrWrrmri num_choicesrtr[ indices_0cls_masknum_non_pad_tokensrrrreshaped_logitsrqrs r7rz#ModernBertForMultipleChoice.forwardsn -6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $** )%'    $AJ ;; ) )U 2 %6%<%rs  $ AA&!JI`9GF&@@A:G   H %z)'z)z299,:BII:( \ 5 \*+B,B4)*N)"))N)+N)b7@\\\~A@/A@A@H >ryy > ? 5?  ? D S *CS  S l 3 '@3  3 l1 %>1 1 h R ";R  R j r8