i+dZddlmZddlmZddlZddlmZddlmZm Z m Z ddl m Z dd lmZdd lmZmZmZdd lmZdd lmZmZdd lmZddlmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5e0jle7Z8GddejrZ: d[dejrdejvdejvdejvdejvdzdeGd#d$ejrZ?Gd%d&ejrZ@Gd'd(ejrZAGd)d*ejrZBGd+d,ejrZCGd-d.eZDGd/d0ejrZEGd1d2ejrZFGd3d4ejrZGGd5d6ejrZHGd7d8ejrZIGd9d:ejrZJGd;de'ZLee/d?@GdAdBe-ZMe/dC@GdDdEeLZNe/dF@GdGdHeLZOe/dI@GdJdKeLeZPe/GdLdMeLZQe/dN@GdOdPeLZRe/dQ@GdRdSeLZSe/GdTdUeLZTe/GdVdWeLZUe/GdXdYeLZVgdZZWy)\zPyTorch BERT model.)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplecheck_model_inputs) BertConfigc eZdZdZfdZ d dej dzdej dzdej dzdejdzded ejf d Z xZ S) BertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0dy) N) padding_idxeps position_idsr&F) persistenttoken_type_ids)dtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr.sizelongselfconfig __class__s p/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/bert/modeling_bert.pyr5zBertEmbeddings.__init__7s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  N input_idsr2r. inputs_embedspast_key_values_lengthreturnc||j}n|jdd}|\}}||jdd|||zf}|t|drT|jj |j dd} t j| d|} | j ||}n:t j|t j|jj}||j|}|j|} || z} |j|} | | z} |j| } |j| } | S)Nr0r2rr&)dimindex)r3device)rIr.hasattrr2rGshaperEgatherrHrJrXr:r>r<r?rC) rLrQr2r.rRrS input_shape batch_size seq_lengthbuffered_token_type_idsr> embeddingsr<s rOforwardzBertEmbeddings.forwardGsP  #..*K',,.s3K!, J  ,,Q0FVlIl0l-lmL  !t-.*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J"%:: "66|D"55 ^^J/ \\*- rP)NNNNr) __name__ __module__ __qualname____doc__r5rE LongTensor FloatTensorintTensorra __classcell__rNs@rOr)r)4sQ $.2260426&' (##d*(((4/(&&- ( ((4/ ( !$ ( (rPr)modulequerykeyvalueattention_maskscalingrCkwargsc ||jddz}tj||jdd|z}|#|ddddddd|jdf}||z}t j j|d}t j j|||j}tj||} | jddj} | |fS) Nr0r rV)ptrainingr&) rIrEmatmul transposerZr functionalsoftmaxrCry contiguous) rlrmrnrorprqrCrr attn_weights attn_outputs rOeager_attention_forwardrrs**R.D(<<s}}Q':;gEL!'1a399R=(@A#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$rPceZdZd fd Z d dej dej dzdedzdej dzdee de ej f d Z xZ S) BertSelfAttentionNc@t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_|j&|_||_||_yNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rt)r4r5r8num_attention_headsrY ValueErrorrMrhattention_head_size all_head_sizerqrLinearrmrnrorAattention_probs_dropout_probrC is_decoder is_causal layer_idxrLrMrrrNs rOr5zBertSelfAttention.__init__sP     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F?*9*N*N'&=%C%C!>2 & "I{)@(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((rPFNNNN rbrcrdr5rErirgr rr!tuplerarjrks@rOrrs}#648(,.2 -)||-)))D0-) -)  t+ -) +, -) u|| -)rPrceZdZd fd Z d dej dej dzdej dzdedzdee de ej f d Z xZ S) BertCrossAttentionNct||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_||_||_yr)r4r5r8rrYrrMrhrrrqrrrmrnrorArrCrrrs rOr5zBertCrossAttention.__init__sC     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \"))#w;FFH L((rPrr) rbrcrdr5rErirgrrr!rrarjrks@rOrrs#4;?376: 2)||2) %00472)))D0 2) -t3 2) +, 2) u|| 2)rPrcneZdZfdZdej dej dej fdZxZS)BertSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr,) r4r5rrr8denser?r@rArBrCrKs rOr5zBertSelfOutput.__init__&s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rPr input_tensorrTcr|j|}|j|}|j||z}|SNrrCr?rLrrs rOrazBertSelfOutput.forward,7 =1  ]3 }|'CD rPrbrcrdr5rErirarjrks@rOrr%1> U\\RWR^R^rPrceZdZd fd Z d dej dej dzdej dzdej dzdedzdej dzd ee d e ej fd Z xZ S) BertAttentionNct|||_|rtnt}|||||_t ||_y)Nrr)r4r5is_cross_attentionrrrLroutput)rLrMrrrattention_classrNs rOr5zBertAttention.__init__4s= "40B,HY#Fi9U $V, rPrrprencoder_attention_maskrrrrrTc |js|n|}|j|f||||d|\}} |j||}|| fS)N)rrprr)rrLr) rLrrprrrrrrattention_outputrs rOrazBertAttention.forward;sg04/F/FLb)2 * "7)+) *  * &, ;;'7G--rP)FNFNNNNNrrks@rOrr3s-48:>;?(,.2.||.))D0. %0047 . !& 1 1D 8 .  . t+.+,. u|| .rPrcVeZdZfdZdej dej fdZxZS)BertIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r4r5rrr8intermediate_sizerr hidden_actstrr intermediate_act_fnrKs rOr5zBertIntermediate.__init__Ss] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rPrrTcJ|j|}|j|}|Sr)rrrLrs rOrazBertIntermediate.forward[s& =1 00? rPrrks@rOrrRs#9U\\ellrPrcneZdZfdZdej dej dej fdZxZS) BertOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r4r5rrrr8rr?r@rArBrCrKs rOr5zBertOutput.__init__bs` YYv779K9KL f&8&8f>S>STzz&"<"<= rPrrrTcr|j|}|j|}|j||z}|Srrrs rOrazBertOutput.forwardhrrPrrks@rOrrarrPrceZdZd fd Z ddej dej dzdej dzdej dzdedzdej dzd ee d e ej fd Z d Z xZ S) BertLayerNct||j|_d|_t ||j ||_|j |_|j|_|jr.|j st|dt |d|d|_ t||_ t||_ y)Nr&rz> should be used as a decoder model if cross attention is addedFT)rrr)r4r5chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rLrMrrNs rOr5zBertLayer.__init__ps '-'E'E$&v9J9JV_` ++#)#=#=  # #?? D6)g!hii"/##' #D  -V4 ( rPrrprrrrrrrTc "|j||f||d|\}} |} |jr:|8t|dstd|d|j|d||fd|i|\} } | } t |j |j|j| } | S)N)rrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrYrrrfeed_forward_chunkrr) rLrrprrrrrrself_attention_output_rcross_attention_output layer_outputs rOrazBertLayer.forwards$24>>  $ ,) $  $  q1 ??4@4!12 =dVDDD )<(;(;%%& ) !0 )  ) % "A 6 0  # #T%A%A4CSCSUe rPcL|j|}|j||}|Sr)rr)rLrintermediate_outputrs rOrzBertLayer.feed_forward_chunks,"//0@A{{#68HI rPrr)rbrcrdr5rErirgr rr!rrarrjrks@rOrros),48:>;?(,.2'||'))D0' %0047 ' !& 1 1D 8 '  ' t+'+,' u|| 'RrPrceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd ej dzd e e d e ej e zfd Z xZS) BertEncoderc t|||_tjt |j Dcgc]}t||c}|_ycc}w)N)r) r4r5rMr ModuleListrangenum_hidden_layersrlayer)rLrMirNs rOr5zBertEncoder.__init__sF  ]]ERXRjRjLk#lqIf$B#lm #lsANrrprrr use_cacherrrrTc t|jD]\} } | |||f|||d|}t||r|SdS)N)rrr)last_hidden_stater) enumeraterr) rLrrprrrrrrrr layer_modules rOrazBertEncoder.forwardsq )4 OA|(%(> /- M 9+/8O  >B  rPNNNNNN)rbrcrdr5rErirgr boolrr!rrrarjrks@rOrrsn48:>;?(,!%.2 || ))D0  %0047  !& 1 1D 8    $;  t+ +,  u|| H H rPrcVeZdZfdZdej dej fdZxZS) BertPoolerct|tj|j|j|_tj |_yr)r4r5rrr8rTanh activationrKs rOr5zBertPooler.__init__s9 YYv1163E3EF '')rPrrTc\|dddf}|j|}|j|}|S)Nr)rr)rLrfirst_token_tensor pooled_outputs rOrazBertPooler.forwards6+1a40 #56  6 rPrrks@rOrrs#$ U\\ellrPrcVeZdZfdZdej dej fdZxZS)BertPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r4r5rrr8rrrrr transform_act_fnr?r@rKs rOr5z$BertPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrPrrTcl|j|}|j|}|j|}|Sr)rr r?rs rOraz#BertPredictionHeadTransform.forwards4 =1 --m< }5 rPrrks@rOrrs$UU\\ellrPrc$eZdZfdZdZxZS)BertLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) r4r5r transformrrr8r7decoder ParameterrErHrrKs rOr5zBertLMPredictionHead.__init__s[ 4V<yy!3!3V5F5FTR LLV->->!?@ rPcJ|j|}|j|}|Sr)rrrs rOrazBertLMPredictionHead.forwards$}5  ]3 rPrbrcrdr5rarjrks@rOr r sArPr cVeZdZfdZdej dej fdZxZS)BertOnlyMLMHeadcBt|t||_yr)r4r5r  predictionsrKs rOr5zBertOnlyMLMHead.__init__s /7rPsequence_outputrTc(|j|}|Sr)r)rLrprediction_scoress rOrazBertOnlyMLMHead.forward s ,,_=  rPrrks@rOrrs#8!u||! !rPrc$eZdZfdZdZxZS)BertOnlyNSPHeadclt|tj|jd|_yNru)r4r5rrr8seq_relationshiprKs rOr5zBertOnlyNSPHead.__init__s'  " &*<*rrr)rPrOr;r;>s~ &*D%  d ")26u((4/68<U..5<59M5**+d2926Je''(4/6rPr;a  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ceZdZddgZdfd ZdZdZee dde jdzde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde dzde dzde jdzde edee jezfdZdZxZS) BertModelr)rct||||_d|_t ||_t ||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r4r5rMgradient_checkpointingr)r`rencoderrpooler post_init)rLrMadd_pooling_layerrNs rOr5zBertModel.__init__fsU   &+#(0"6* ,=j(4  rPc.|jjSrr`r:rLs rOget_input_embeddingszBertModel.get_input_embeddingsws...rPc&||j_yrrJ)rLros rOset_input_embeddingszBertModel.set_input_embeddingszs*/'rPNrQrpr2r.rRrrrrrrrrTc L|jjr| | n|jj} nd} | rd|b||jjr4t t |jt |jnt |j}|du|duz r t d||j} |jd} n|j} |jd} ||jnd}| tj||| z| } |j|||||}|j||||| |\}}|j|f||||| | |d | }|j}|j |j!|nd}t#|||j$ S) NF)rMz:You must specify exactly one of input_ids or inputs_embedsr&r)rX)rQr.r2rRrS)rprembedding_outputrrr)rprrrrrr.)r pooler_outputr)rMrris_encoder_decoderrr rrXrZget_seq_lengthrErFr`_create_attention_masksrErrFrr)rLrQrpr2r.rRrrrrrrrrXr^rSrPencoder_outputsrrs rOrazBertModel.forward}s ;; ! !%.%:  @U@UII 0)4 8V8V$L $DlZ^ZeZeFfg!5   -t"; <YZ Z  %%F"+J"))F&,,Q/JETE`!?!?!Afg  !"\\*@BX[eBentuN??%)'#9 + 261M1M)#9-"7)+ 2N2 ..'$,,   )"7#9+)%    *;;8< 8O O4UY ;-'+;;  rPc|jjrt|j||||}nt|j||}|t|j|||}||fS)N)rM input_embedsrprr)rMrWrp)rMrWrpr)rMrrr)rLrprrPrrrs rOrTz!BertModel._create_attention_maskssx ;; ! !/{{--- / N7{{--N " -%>{{-5&; & "555rP)T) NNNNNNNNNN)rbrcrd_no_split_modulesr5rLrNr%r"rErir rrr!rrrarTrjrks@rOrBrBWsJ*;7"/0*..2.2,0-1596:(,!%.2K <<$&K  t+K  t+ K llT) K ||d* K  %||d2K !& t 3K K $;K  t+K +,K  u|| K KK K Z 6rPrBz Bert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c`eZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde e de e jezfdZxZS)BertForPreTraining&bert.embeddings.word_embeddings.weightcls.predictions.biaszcls.predictions.decoder.weightzcls.predictions.decoder.biasct||t||_t ||_|j yr)r4r5rBr(r#clsrGrKs rOr5zBertForPreTraining.__init__s4  f% '/ rPcB|jjjSrr_rrrKs rOget_output_embeddingsz(BertForPreTraining.get_output_embeddingsxx##+++rPc||jj_|j|jj_yrr_rrrrLnew_embeddingss rOset_output_embeddingsz(BertForPreTraining.set_output_embeddings,'5$$2$7$7!rPNrQrpr2r.rRlabelsnext_sentence_labelrrrTc |j|f||||dd|} | dd\} } |j| | \} } d}|u|st}|| jd|jj |jd}|| jdd|jd}||z}t || | | j| jS)am labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForPreTraining.from_pretrained("google-bert/bert-base-uncased") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Trpr2r.rR return_dictNrur0)r<r=r>rr)) r(r_rrrMr7r;rr))rLrQrpr2r.rRrjrkrroutputsrrrr! total_lossloss_fctmasked_lm_lossnext_sentence_losss rOrazBertForPreTraining.forward sR$))  ))%'   *1!&48HH_m4\11  "5"A')H%&7&<& ,8*..2.2,0-1&*37A <<$&A  t+A  t+ A llT) A ||d* A  t#A #\\D0A +,A  u|| 7 7A A rPrZzP Bert Model with a `language modeling` head on top for CLM fine-tuning. c eZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde dzde dzde jdzde e jzdeedee jezfdZxZS)BertLMHeadModelr[r\r]ct|||jstj dt |d|_t||_|jy)NzLIf you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`FrH r4r5rloggerwarningrBr(rr_rGrKs rOr5zBertLMHeadModel.__init__\sL    NNi jf> "6* rPcB|jjjSrrarKs rOrbz%BertLMHeadModel.get_output_embeddingshrcrPc||jj_|j|jj_yrrerfs rOrhz%BertLMHeadModel.set_output_embeddingskrirPNrQrpr2r.rRrrrjrrrlogits_to_keeprrrTc  |d} |j|f||||||| | | dd | }|j}t| tr t | dn| }|j |dd|ddf}d}|*|j d|||jjd| }t|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NFT) rpr2r.rRrrrrrrn)logitsrjr7)r<rrrr)r*r@)r(rrrhslicer_ loss_functionrMr7rrrr)r*)rLrQrpr2r.rRrrrjrrrrrrror slice_indicesrr<s rOrazBertLMHeadModel.forwardos0  I@I  A ))%'"7#9+) A  A  11 8B>SV8W~ot4]k -=!(;<=  %4%%pVFt{{OeOepiopD0#33!//))$55   rP) NNNNNNNNNNNr)rbrcrdrur5rbrhr$r"rErir rrhrr!rrrarjrks@rOrwrwQsf+S(> ,8*..2.2,0-1596:&*(,!%.2-.8 <<$&8  t+8  t+ 8 llT) 8 ||d* 8  %||d28 !& t 38  t#8 8 $;8  t+8 ell*8 +,8  u|| @ @8 8 rPrwceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde e de e jezfdZxZS)BertForMaskedLMr[r\r]ct|||jrtj dt |d|_t||_|jy)NzkIf you want to use `BertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FryrzrKs rOr5zBertForMaskedLM.__init__sR     NN1  f> "6* rPcB|jjjSrrarKs rOrbz%BertForMaskedLM.get_output_embeddingsrcrPc||jj_|j|jj_yrrerfs rOrhz%BertForMaskedLM.set_output_embeddingsrirPNrQrpr2r.rRrrrjrrrTc >|j|f||||||dd| } | d} |j| } d} |Ft}|| jd|jj |jd} t | | | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rpr2r.rRrrrnrNr0r<rrr)) r(r_rrrMr7rrr))rLrQrpr2r.rRrrrjrrrorrrrrqs rOrazBertForMaskedLM.forwards($))   ))%'"7#9    "!* HH_5  ')H%&7&<& ,8*..2.2,0-1596:&*+ <<$&+  t++  t+ + llT) + ||d* +  %||d2+ !& t 3+  t#+ +,+  u|| ~ -+ + rPrzT Bert Model with a `next sentence prediction (classification)` head on top. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForNextSentencePredictionct||t||_t ||_|j yr)r4r5rBr(rr_rGrKs rOr5z&BertForNextSentencePrediction.__init__s4  f% "6* rPNrQrpr2r.rRrjrrrTc |j|f||||dd|}|d} |j| } d} |2t} | | jdd|jd} t | | |j |j S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForNextSentencePrediction.from_pretrained("google-bert/bert-base-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trmr&Nr0rur)r(r_rrrrr)) rLrQrpr2r.rRrjrrrorseq_relationship_scoresrsrqs rOraz%BertForNextSentencePrediction.forwardsN$))  ))%'     "&((="9!  ')H!)*A*F*Fr1*Mv{{[]!_ *#*!//))   rPr)rbrcrdr5r$r"rErirr!rrrarjrks@rOrrs *..2.2,0-1&*= <<$&=  t+=  t+ = llT) = ||d* =  t#= +,=  u|| : := = rPrz Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForSequenceClassificationcnt|||j|_||_t ||_|j |j n |j}tj||_ tj|j|j|_ |jyr)r4r5 num_labelsrMrBr(classifier_dropoutrBrrArCrr8 classifierrGrLrMrrNs rOr5z&BertForSequenceClassification.__init__Qs   ++ f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rPNrQrpr2r.rRrjrrrTc |j|f||||dd|}|d} |j| } |j| } d} ||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt} |j dk(r&| | j|j} n| | |} n|jjdk(r=t} | | jd|j |jd} n,|jjdk(rt} | | |} t| | |j |j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trmr&N regressionsingle_label_classificationmulti_label_classificationr0r)r(rCrrM problem_typerr3rErJrhrsqueezerrrrrr)) rLrQrpr2r.rRrjrrrorrr<rqs rOraz%BertForSequenceClassification.forward`s$$))  ))%'      ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   rPr)rbrcrdr5r$r"rErirr!rrrarjrks@rOrrJs *..2.2,0-1&*; <<$&;  t+;  t+ ; llT) ; ||d* ;  t#; +,;  u|| 7 7; ; rPrc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForMultipleChoicec*t||t||_|j |jn |j }t j||_t j|jd|_ |jy)Nr&) r4r5rBr(rrBrrArCrr8rrGrs rOr5zBertForMultipleChoice.__init__su  f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rPNrQrpr2r.rRrjrrrTc ||jdn|jd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f||||dd|} | d} |j | } |j | } | jd|} d} |t }|| |} t| | | j| jS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr&r0rvTrmr) rZrrIr(rCrrrrr))rLrQrpr2r.rRrjrr num_choicesrorrreshaped_logitsr<rqs rOrazBertForMultipleChoice.forwardsT-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $))  ))%'      ]3 / ++b+6  ')HOV4D("!//))   rPr)rbrcrdr5r$r"rErirr!rrrarjrks@rOrrs *..2.2,0-1&*N <<$&N  t+N  t+ N llT) N ||d* N  t#N +,N  u|| 8 8N N rPrc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyNFry)r4r5rrBr(rrBrrArCrr8rrGrs rOr5z#BertForTokenClassification.__init__s   ++f> )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rPNrQrpr2r.rRrjrrrTc H|j|f||||dd|}|d} |j| } |j| } d} | ))F$6$68I8IJ rPNrQrpr2r.rRstart_positions end_positionsrrrTc |j|f||||dd|} | d} |j| } | jdd\} } | jdj } | jdj } d}||t |j dkDr|jd}t |j dkDr|jd}| j d}|jd|}|jd|}t|}|| |}|| |}||zdz }t|| | | j| j S) NTrmrr&r0rw) ignore_indexru)r< start_logits end_logitsrr)) r(rsplitrr~lenrIclamprrrr))rLrQrpr2r.rRrrrrrorrrrrp ignored_indexrq start_lossend_losss rOraz BertForQuestionAnswering.forwardJs$))  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   rPrt)rbrcrdr5r$r"rErirr!rrrarjrks@rOrr>s*..2.2,0-1/3-13 <<$&3  t+3  t+ 3 llT) 3 ||d* 3 ,3 ||d*3 +,3  u|| ; ;3 3 rPr) rrrrZrrrrrwrBr')Nr)Xrecollections.abcr dataclassesrrErtorch.nnrrrr r- activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsr r!r"r# utils.genericr$r%configuration_bertr' get_loggerrbr{Moduler)rifloatrrrrrrrrrrrr rrr#r'r;rBrZrwrrrrrr__all__r@rPrOrs$! AA&!CC)J9   G&6MMA*   H %;RYY;H!% II% <<% % << % LL4' % T\ %%'(%:F) F)RJ)J)Z RYY .BII.> ryy   @*@F ")) D  "))" 299  !bii!&bii& 9299 9////2  7{7  7&  I6#I6 I6X Y ,Y  Y x S )?S  S lI )I I X I $7I  I X M $7M  M `^ /^ ^ B8 !48 8 v@ 2@ @ F rP