iddlmZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*e%jVe,Z-Gddej\Z/ej`e/dZ1Gddej\Z2 d^dej\dejfdejfdejfdejfdzde4dzde4de e#fdZ5Gd d!ej\Z6Gd"d#ej\Z7Gd$d%ej\Z8Gd&d'ej\Z9Gd(d)ej\Z:Gd*d+ej\Z;Gd,d-ej\Z<Gd.d/ej\Z=Gd0d1ej\Z>Gd2d3ej\Z?Gd4d5eZ@Gd6d7ej\ZAGd8d9ej\ZBGd:d;ej\ZCGd<d=ej\ZDGd>d?ej\ZEGd@dAej\ZFe$GdBdCeZGee$dDEGdFdGe"ZHe$GdHdIeGZIe$dJEGdKdLeGZJe$GdMdNeGZKGdOdPej\ZLe$dQEGdRdSeGZMe$dTEGdUdVeGZNe$GdWdXeGZOe$GdYdZeGZPe$Gd[d\eGZQgd]ZRy)_)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplecheck_model_inputs)MobileBertConfigcXeZdZdfd Zdej dej fdZxZS)NoNormct|tjt j ||_tjt j||_yN) super__init__r Parametertorchzerosbiasonesweight)self feat_sizeeps __class__s |/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/mobilebert/modeling_mobilebert.pyr&zNoNorm.__init__7s@ LLY!78 ll5::i#89 input_tensorreturnc:||jz|jzSr$)r,r*)r-r3s r1forwardzNoNorm.forward<sdkk)DII55r2r$__name__ __module__ __qualname__r&r(Tensorr6 __classcell__r0s@r1r"r"6s#: 6ELL6U\\6r2r") layer_normno_normc eZdZdZfdZ d dej dzdej dzdej dzdejdzdejf d Z xZ S) MobileBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cXt||j|_|j|_|j|_t j |j|j|j|_ t j |j|j|_ t j |j|j|_ |jrdnd}|j|z}t j||j|_t!|j"|j|_t j&|j(|_|j-dt/j0|jj3ddy)N) padding_idxr r position_idsrF) persistent)r%r& trigram_inputembedding_size hidden_sizer Embedding vocab_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddingsLinearembedding_transformationNORM2FNnormalization_type LayerNormDropouthidden_dropout_probdropoutregister_bufferr(arangeexpand)r-configembed_dim_multiplierembedded_input_sizer0s r1r&zMobileBertEmbeddings.__init__FsF #11$33!--!||F,=,=v?T?Tbhbubuv#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"$($6$6qA"114HH(* 2EvGYGY(Z% !:!:;F J"%88;PP ^^J/ \\*- r2)NNNN) r8r9r:__doc__r&r( LongTensor FloatTensorr;r6r<r=s@r1rArACs~Q 0.2260426 0##d*0((4/0&&- 0 ((4/ 0  0r2rAmodulequerykeyrhattention_maskscalingrZkwargsc ||jddz}tj||jdd|z}|#|ddddddd|jdf}||z}t j j|d}t j j|||j}tj||} | jddj} | |fS) NrFrir rj)ptrainingr) rlr(matmul transposeshaperrosoftmaxrZr contiguous) rwrxryrhrzr{rZr| attn_weights attn_outputs r1eager_attention_forwardrs**R.D(<<s}}Q':;gEL!'1a399R=(@A#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r2ceZdZfdZ d dej dej dej dej dzdeede ej f d Z xZ S) MobileBertSelfAttentionct|||_|j|_t |j |jz |_|j|j z|_|j dz|_tj|j |j|_ tj|j |j|_ tj|jr |j n |j|j|_tj |j"|_d|_y)Nr~F)r%r&r^num_attention_headsinttrue_hidden_sizeattention_head_size all_head_sizer{rrSrxryuse_bottleneck_attentionrJrhrXattention_probs_dropout_probrZ is_causalr-r^r0s r1r&z MobileBertSelfAttention.__init__s  #)#=#= #&v'>'>A[A['[#\ !558P8PP//5 YYv668J8JK 99V44d6H6HIYY'-'F'FF # #FL^L^`d`r`r  zz&"E"EF r2N query_tensor key_tensor value_tensorrzr|r4c |jdd}g|d|j}|j|j|j dd}|j |j|j dd} |j |j|j dd} tj|jjt} | ||| | |f|jsdn|jj|jd|\} } | j g|dj#} | | fS)NrFrrirg)rZr{)rrrxviewrryrhr get_interfacer^_attn_implementationrrrZrr{reshaper)r-rrrrzr|rq hidden_shape query_layer key_layer value_layerattention_interfacerrs r1r6zMobileBertSelfAttention.forwardsH#(("- CCbC$*B*BC 4djj.33\BLLQPQR -DHHZ(--|<FFq!L 3djj.33\BLLQPQR (?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((r2r$ r8r9r:r&r(r;rvrrtupler6r<r=s@r1rrsr,48 )ll)LL)ll ) ))D0 ) +, ) u|| )r2rcneZdZfdZdej dej dej fdZxZS)MobileBertSelfOutputcjt||j|_tj|j |j |_t|j|j |j|_ |js%tj|j|_ yyNr/)r%r&use_bottleneckrrSrdenserUrVlayer_norm_epsrWrXrYrZrs r1r&zMobileBertSelfOutput.__init__s $33YYv668O8OP  !:!:;F U\\ELLUZUaUar2rceZdZfdZdej dej dej dej fdZxZS)MobileBertOutputcrt||j|_tj|j |j |_t|j|j |_ |js%tj|j|_ yt||_yr$)r%r&rrrSrrrrUrVrWrXrYrZr bottleneckrs r1r&zMobileBertOutput.__init__,s $33YYv779P9PQ  !:!:;F>,9J*JKL >>,9J*JKL??<9JKLr2r7r=s@r1rr+s?7 #(<< DILL ejeqeq   r2rcVeZdZfdZdej dej fdZxZS)BottleneckLayerct|tj|j|j |_t|j|j |j|_ yr) r%r&rrSrJintra_bottleneck_sizerrUrVrrWrs r1r&zBottleneckLayer.__init__DsR YYv1163O3OP  !:!:;F"  ( (.014 4  - -%)^^M%B "*,BMSmn n!=-A[\ \r2 r8r9r:r&r(r;rr6r<r=s@r1rrOs+5]U\\]eELL6I]r2rcneZdZfdZdej dej dej fdZxZS) FFNOutputct|tj|j|j |_t|j|j |j|_ yr) r%r&rrSrrrrUrVrrWrs r1r&zFFNOutput.__init__tsR YYv779P9PQ  !:!:;F $7G r2r7r=s@r1rrs#( U\\ellr2rc eZdZfdZ ddej dej dzdeede ej fdZ xZ S) MobileBertLayerct||j|_|j|_t ||_t ||_t||_ |jrt||_ |jdkDrHtjt|jdz Dcgc] }t|c}|_yycc}wNr)r%r&rnum_feedforward_networksrrrrrrrrr ModuleListrangerffnr-r^_r0s r1r&zMobileBertLayer.__init__s $33(.(G(G%,V426:&v.   (0DO  * *Q .}}fFeFehiFi@j%k1hv&6%klDH /%ks6CNrrzr|r4c 2|jr|j|\}}}}n |gdz\}}}}|j|||||fi|\}} |} |jdk7r|jD] } | | } |j | } |j | | |} | S)Nrr)rrrrrrr)r-rrzr|rrrrself_attention_outputrr ffn_modulerrs r1r6zMobileBertLayer.forwards   BF//R_B` ?L*lKCP/TUBU ?L*lK#14>>      $  $  q1  ( (A -"hh @ #-.>#?  @#//0@A{{#68H-X r2r$rr=s@r1rrsW m 48||))D0+,  u||  r2rc teZdZfdZ ddej dej dzdeede e zfdZ xZ S) MobileBertEncoderct|tjt |j Dcgc] }t |c}|_ycc}wr$)r%r&rrrnum_hidden_layersrlayerrs r1r&zMobileBertEncoder.__init__s< ]]U6KcKcEd#eOF$;#ef #esANrrzr|r4c ht|jD]\}}|||fi|}t|S)N)last_hidden_state) enumeraterr)r-rrzr|i layer_modules r1r6zMobileBertEncoder.forwardsG )4 OA|(M  ??r2r$) r8r9r:r&r(r;rvrrrrr6r<r=s@r1rrsXg48 @|| @))D0 @+, @  @r2rcVeZdZfdZdej dej fdZxZS)MobileBertPoolerct||j|_|jr0t j |j |j |_yyr$)r%r&classifier_activation do_activaterrSrJrrs r1r&zMobileBertPooler.__init__sH !77   6#5#5v7I7IJDJ r2rr4c|dddf}|js|S|j|}tj|}|S)Nr)rrr(tanh)r-rfirst_token_tensor pooled_outputs r1r6zMobileBertPooler.forwardsE+1a40% % JJ'9:M!JJ}5M r2r7r=s@r1rrs$K !U\\ !ell !r2rcVeZdZfdZdej dej fdZxZS)!MobileBertPredictionHeadTransformcZt|tj|j|j|_t |jtrt|j|_ n|j|_ td|j|j|_ y)Nr>r)r%r&rrSrJrrrrr transform_act_fnrUrrWrs r1r&z*MobileBertPredictionHeadTransform.__init__s| YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D ! .v/A/AvG\G\]r2rr4cl|j|}|j|}|j|}|Sr$)rrrWrs r1r6z)MobileBertPredictionHeadTransform.forwards4 =1 --m< }5 r2r7r=s@r1r r s$^U\\ellr2r cVeZdZfdZdej dej fdZxZS)MobileBertLMPredictionHeadct|t||_t j |j |j|jz d|_ t j |j|j d|_ t jtj|j |_y)NF)r*T)r%r&r  transformrrSrLrJrIrdecoderr'r(r)r*rs r1r&z#MobileBertLMPredictionHead.__init__s :6BYYv00&2D2DvG\G\2\chi yy!6!68I8IPTU LLV->->!?@ r2rr4c|j|}|jtj|jj j |jj gd}||jjz }|S)Nrrj) rrr(rnrr,trr*rs r1r6z"MobileBertLMPredictionHead.forwardsk}5 %,,UYY 8K8K8M8M8OQUQ[Q[QbQb7cij-kl *** r2r7r=s@r1rrs$AU\\ellr2rcVeZdZfdZdej dej fdZxZS)MobileBertOnlyMLMHeadcBt|t||_yr$)r%r&r predictionsrs r1r&zMobileBertOnlyMLMHead.__init__s 5f=r2sequence_outputr4c(|j|}|Sr$)r)r-rprediction_scoress r1r6zMobileBertOnlyMLMHead.forwards ,,_=  r2r7r=s@r1rrs#>!u||! !r2rcteZdZfdZdej dej deej fdZxZS)MobileBertPreTrainingHeadsct|t||_t j |j d|_yNri)r%r&rrrrSrJseq_relationshiprs r1r&z#MobileBertPreTrainingHeads.__init__ s4 5f= " &*<*%> HH & &~RV&?& "w.n.MMr2rarzrbrDrclabelsnext_sentence_labelr|c |j|f||||dd|} | dd\} } |j| | \} } d}|u|st}|| jd|jj |jd}|| jdd|jd}||z}t || | | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForPreTraining.from_pretrained("google/mobilebert-uncased") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```TrzrbrDrc return_dictNrirF)r;r<r=rr() r'rYrrr^rLr:rr()r-rarzrbrDrcrlrmr|outputsrr rr$ total_lossloss_fctmasked_lm_lossnext_sentence_losss r1r6z MobileBertForPreTraining.forwardsR"$//  ))%'   *1!&48HH_m4\11  "5"A')H%&7&<&%> HH & &~RV&?& "w.n.MMr2rarzrbrDrcrlr|c :|j|f||||dd|}|d} |j| } d} |Ft} | | jd|jj |jd} t | | |j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` TrorNrFr;logitsrr() r'rYrrr^rLrrr() r-rarzrbrDrcrlr|rqrrrtrss r1r6zMobileBertForMaskedLM.forwards$"$//  ))%'   "!* HH_5  ')H%&7&<&>> from transformers import AutoTokenizer, MobileBertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForNextSentencePrediction.from_pretrained("google/mobilebert-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> loss = outputs.loss >>> logits = outputs.logits ```TrorNrFrir)r'rYrrrrr() r-rarzrbrDrcrlr|rqr r$rurss r1r6z+MobileBertForNextSentencePrediction.forward[sL"$//  ))%'     !%-!8!  ')H!)*@*E*Eb!*LfkkZ\o!^ *#)!//))   r2r)r8r9r:r&rrr(rurvrrrrr6r<r=s@r1rrLs .237260426*.; ##d*; ))D0; ((4/ ; &&- ; ((4/ ;   4'; +,;  , ,; ; r2rz MobileBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) #MobileBertForSequenceClassificationcnt|||j|_||_t ||_|j |j n |j}tj||_ tj|j|j|_ |jyr$)r%r& num_labelsr^r@r'classifier_dropoutrYrrXrZrSrJ classifierrEr-r^rr0s r1r&z,MobileBertForSequenceClassification.__init__s   ++ )&1)/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r2NrarzrbrDrcrlr|r4c |j|f||||dd|}|d} |j| } |j| } d} ||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt} |j dk(r&| | j|j} n| | |} n|jjdk(r=t} | | jd|j |jd} n,|jjdk(rt} | | |} t| | |j |j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). TrorN regressionsingle_label_classificationmulti_label_classificationrFr)r'rZrr^ problem_typerrer(rmrrsqueezerrrrrr() r-rarzrbrDrcrlr|rqr rr;rss r1r6z+MobileBertForSequenceClassification.forwards$"$//  ))%'      ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r2r)r8r9r:r&rrr(r;rrrrr6r<r=s@r1rrs *..2.2,0-1&*; <<$&;  t+;  t+ ; llT) ; ||d* ;  t#; +,;  u|| 7 7; ; r2rcJeZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)MobileBertForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyr{) r%r&rr@r'rrSrJ qa_outputsrErs r1r&z'MobileBertForQuestionAnswering.__init__sU   ++)&EJ))F$6$68I8IJ r2NrarzrbrDrcstart_positions end_positionsr|r4c |j|f||||dd|} | d} |j| } | jdd\} } | jdj } | jdj } d}||t |j dkDr|jd}t |j dkDr|jd}| j d}|jd|}|jd|}t|}|| |}|| |}||zdz }t|| | | j| j S) NTrorrrFrj) ignore_indexri)r; start_logits end_logitsrr() r'rsplitrrlenrlclamprrrr()r-rarzrbrDrcrrr|rqrrrrrr ignored_indexrs start_lossend_losss r1r6z&MobileBertForQuestionAnswering.forwards"$//  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   r2rv)r8r9r:r&rrr(r;rrrrr6r<r=s@r1rrs*..2.2,0-1/3-13 <<$&3  t+3  t+ 3 llT) 3 ||d* 3 ,3 ||d*3 +,3  u|| ; ;3 3 r2rc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) MobileBertForMultipleChoicec*t||t||_|j |jn |j }t j||_t j|jd|_ |jyr) r%r&r@r'rrYrrXrZrSrJrrErs r1r&z$MobileBertForMultipleChoice.__init__:su  )&1)/)B)B)NF % %TZTnTn zz"45 ))F$6$6: r2NrarzrbrDrcrlr|r4c ||jdn|jd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f||||dd|} | d} |j | } |j | } | jd|} d} |t }|| |} t| | | j| jS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NrrFrTror) rrrlr'rZrrrrr()r-rarzrbrDrcrlr| num_choicesrqr rreshaped_logitsr;rss r1r6z#MobileBertForMultipleChoice.forwardGsT-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  "$//  ))%'      ]3 / ++b+6  ')HOV4D("!//))   r2r)r8r9r:r&rrr(r;rrrrr6r<r=s@r1rr7s *..2.2,0-1&*N <<$&N  t+N  t+ N llT) N ||d* N  t#N +,N  u|| 8 8N N r2rc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) MobileBertForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyr{)r%r&rr@r'rrYrrXrZrSrJrrErs r1r&z)MobileBertForTokenClassification.__init__s   ++)&EJ)/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r2NrarzrbrDrcrlr|r4c H|j|f||||dd|}|d} |j| } |j| } d} |r2r1rs.%! AA&!69   G&MMA6   H %6RYY6& 9I299If!% II% <<% % << % LL4' % T\ %%'(%:1)bii1)h299".")).8 RYY  ryy ryy0 bii !]!]H   ryy )0)X@ @(!ryy!& ""!BII! 9 9iii4  7[7  7&B /B B J _ 8_  _ DF 5F F R&BII& G *CG  G TM *CM M `@ %>@ @ F^ ";^ ^ B8 '@8 8 v r2