iDdZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z dd lmZdd lmZmZmZmZmZmZdd lmZmZdd lmZdd lmZmZmZddl m!Z!ejDe#Z$GddejJZ&GddejJZ'GddejJZ(GddejJZ)GddejJZ*GddejJZ+GddejJZ,GddejJZ-Gd d!ejJZ. d@d"ejJd#ej^d$ej^d%ej^d&ej^dzd'e0d(e0fd)Z1Gd*d+ejJZ2Gd,d-ejJZ3Gd.d/eZ4Gd0d1ejJZ5eGd2d3eZ6eGd4d5e6Z7eGd6d7e6Z8ed89Gd:d;e6Z9ed<9Gd=d>e6Z:gd?Z;y)AzPyTorch MarkupLM model.)CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward)auto_docstringcan_return_tuplelogging)MarkupLMConfigc*eZdZdZfdZddZxZS)XPathEmbeddingszConstruct the embeddings from xpath tags and subscripts. We drop tree-id in this version, as its info can be covered by xpath. c t||j|_tj|j |jz|j |_tj|j|_ tj|_ tj|j |jzd|j z|_ tjd|j z|j |_tjt!|jDcgc],}tj"|j$|j .c}|_tjt!|jDcgc],}tj"|j(|j .c}|_ycc}wcc}w)N)super__init__ max_depthrLinearxpath_unit_hidden_size hidden_sizexpath_unitseq2_embeddingsDropouthidden_dropout_probdropoutReLU activationxpath_unitseq2_inner inner2emb ModuleListrange Embeddingmax_xpath_tag_unit_embeddingsxpath_tag_sub_embeddingsmax_xpath_subs_unit_embeddingsxpath_subs_sub_embeddings)selfconfig_ __class__s x/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/markuplm/modeling_markuplm.pyrzXPathEmbeddings.__init__0sZ )))+63P3PSWSaSa3acicucu)v&zz&"<"<= '')$&IIf.K.Kdnn.\^_bhbtbt^t$u!1v'9'9#96;M;MN(* t~~.  VAA6C`C`a ) %*,t~~.  VBBFDaDab * &   s 11G1Gc g}g}t|jD]^}|j|j||dddd|f|j|j||dddd|f`t j |d}t j |d}||z}|j|j|j|j|}|S)Ndim) r-r appendr0r2torchcatr+r'r)r*)r3xpath_tags_seqxpath_subs_seqxpath_tags_embeddingsxpath_subs_embeddingsixpath_embeddingss r7forwardzXPathEmbeddings.forwardJs " "t~~& eA ! ( ()I)F)Fq)I.YZ\]_`Y`Ja)b c ! ( ()J)G)G)J>Z[]^`aZaKb)c d e!& *?R H % *?R H03HH>>$,,ttG`G`aqGr7s*tu)NN)__name__ __module__ __qualname____doc__rrE __classcell__r6s@r7rr*s  4 rFrcXeZdZdZfdZedZeddZ ddZxZ S)MarkupLMEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cdt|||_tj|j |j |j|_tj|j|j |_ |j|_ t||_ tj|j|j |_tj |j |j"|_tj$|j&|_|j+dt-j.|jj1dd|j|_tj|j|j |j2|_ y)N) padding_idxeps position_idsrr9F) persistent)rrr4rr. vocab_sizer# pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsr rrDtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsr%r&r'register_bufferr=arangeexpandrPr3r4r6s r7rzMarkupLMEmbeddings.__init___s:  !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c )) / 7%'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  "..#%<<  * *F,>,>DL\L\$  rFc|jdd}|d}tj|dz||zdztj|j}|j dj |S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr9rdtypedevicer)sizer=r`longrf unsqueezera) inputs_embedsrP input_shapesequence_lengthrSs r7&create_position_ids_from_inputs_embedsz9MarkupLMEmbeddings.create_position_ids_from_inputs_embedswsp$((*3B/ %a.|| !O_{:Q>ejjYfYmYm %%a(// <>*::;5::V\7N  !![[44uzzd;'4>>*::;5::V\8N)"66|D $ : :> J00P%(;;>SSVff ^^J/ \\*- rF)r)NNNNNN) rGrHrIrJr staticmethodrmrwrErKrLs@r7rNrN\sLQ 0==" 8 8"1rFrNcneZdZfdZdej dej dej fdZxZS)MarkupLMSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNrQ) rrrr!r#denser]r^r%r&r'rbs r7rzMarkupLMSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rF hidden_states input_tensorreturncr|j|}|j|}|j||z}|SNrr'r]r3rrs r7rEzMarkupLMSelfOutput.forward7 =1  ]3 }|'CD rFrGrHrIrr=TensorrErKrLs@r7rr1> U\\RWR^R^rFrcVeZdZfdZdej dej fdZxZS)MarkupLMIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) rrrr!r#intermediate_sizer isinstance hidden_actstrr intermediate_act_fnrbs r7rzMarkupLMIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rFrrcJ|j|}|j|}|Sr)rrr3rs r7rEzMarkupLMIntermediate.forwards& =1 00? rFrrLs@r7rrs#9U\\ellrFrcneZdZfdZdej dej dej fdZxZS)MarkupLMOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) rrrr!rr#rr]r^r%r&r'rbs r7rzMarkupLMOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rFrrrcr|j|}|j|}|j||z}|Srrrs r7rEzMarkupLMOutput.forwardrrFrrLs@r7rrrrFrcVeZdZfdZdej dej fdZxZS)MarkupLMPoolerct|tj|j|j|_tj |_yr)rrrr!r#rTanhr)rbs r7rzMarkupLMPooler.__init__s9 YYv1163E3EF '')rFrrc\|dddf}|j|}|j|}|S)Nr)rr))r3rfirst_token_tensor pooled_outputs r7rEzMarkupLMPooler.forwards6+1a40 #56  6 rFrrLs@r7rrs#$ U\\ellrFrcVeZdZfdZdej dej fdZxZS)MarkupLMPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) rrrr!r#rrrrr transform_act_fnr]r^rbs r7rz(MarkupLMPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrFrrcl|j|}|j|}|j|}|Sr)rrr]rs r7rEz'MarkupLMPredictionHeadTransform.forwards4 =1 --m< }5 rFrrLs@r7rrs$UU\\ellrFrc$eZdZfdZdZxZS)MarkupLMLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) rrr transformrr!r#rVdecoder Parameterr=ryrrbs r7rz!MarkupLMLMPredictionHead.__init__!s[ 8@yy!3!3V5F5FTR LLV->->!?@ rFcJ|j|}|j|}|Sr)rrrs r7rEz MarkupLMLMPredictionHead.forward*s$}5  ]3 rF)rGrHrIrrErKrLs@r7rr sArFrcVeZdZfdZdej dej fdZxZS)MarkupLMOnlyMLMHeadcBt|t||_yr)rrr predictionsrbs r7rzMarkupLMOnlyMLMHead.__init__2s 3F;rFsequence_outputrc(|j|}|Sr)r)r3rprediction_scoress r7rEzMarkupLMOnlyMLMHead.forward6s ,,_=  rFrrLs@r7rr1s#<!u||! !rFrmodulequerykeyvalueattention_maskscalingr'c tj||jdd|z}|#|ddddddd|jdf} || z}tj j |dtjj|j}tj j|||j}tj||} | jddj} | |fS)Nrr9)r;re)ptrainingr) r=matmul transposeshaper functionalsoftmaxfloat32torer'r contiguous) rrrrrrr'kwargs attn_weights causal_mask attn_outputs r7eager_attention_forwardr<s<<s}}Q':;gEL!$Q1o " o%=> #k1 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$rFc eZdZfdZ ddej dej dzdedzdeej fdZ xZ S) MarkupLMSelfAttentionc$t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ tj|j|j|_ tj|j|j|_ tj|j|j|_tj|j |_|j |_|jdz|_y)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()g)rrr#num_attention_headshasattr ValueErrorr4rpattention_head_size all_head_sizerr!rrrr%attention_probs_dropout_probr'attention_dropoutrrbs r7rzMarkupLMSelfAttention.__init__UsC     : : :a ?PVXhHi#F$6$6#78 445Q8   #)#=#= #&v'9'9F>> from transformers import AutoProcessor, MarkupLMModel >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = MarkupLMModel.from_pretrained("microsoft/markuplm-base") >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 768] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer9z5You have to specify either input_ids or inputs_embeds)rfrdrr)reg?g)rsr?r@rSrrjT)rr r r)r pooler_outputrr)r4rr use_return_dictr%warn_if_padding_and_no_attention_maskrgrfr=r{ryrhrirrerr%r&r rr)r3rsr?r@rrrSrjrr r rrkrfextended_attention_maskembedding_outputencoder_outputsrrs r7rEzMarkupLMModel.forward"sN2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U%.%:!! @T@T  !"ZZ FCN  !"[[EJJvVN"0":":1"="G"G"J"9"<"<4::"<"N#&)@#@H"L??))%)' + ,,  #/!5 ' *!,8< 8O O4UY )-')77&11   rF)T) NNNNNNNNNN)rGrHrIrr+r-rrr= LongTensorrrr|r rErKrLs@r7r#r# s& /0.2262637260426)-,0#'X ##d*X ((4/X ((4/ X ))D0 X ((4/ X &&-X ((4/X  $;X #TkX D[X  + +X X rFr#ceZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd edzd edzdedzde eje zfdZ xZ S)MarkupLMForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyNF)r() rr num_labelsr#rrr!r# qa_outputsr'rbs r7rz%MarkupLMForQuestionAnswering.__init__sU   ++%fF ))F$6$68I8IJ rFNrsr?r@rrrSrjstart_positions end_positionsrr r rc | | n|jj} |j|||||||| | d }|d}|j|}|j dd\}}|j dj }|j dj }d}|| t|jdkDr|j d}t| jdkDr| j d} |jd}|jd|| jd|t|}|||}||| }||zd z }t||||j|j S) ae xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMForQuestionAnswering >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> html_string = " My name is Niels " >>> question = "What's his name?" >>> encoding = processor(html_string, questions=question, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> processor.decode(predict_answer_tokens).strip() 'Niels' ```NT r?r@rrrSrjrr r rrr9r:) ignore_indexr)loss start_logits end_logitsrr)r4r0rr;splitsqueezerlenrgclamp_rrrr)r3rsr?r@rrrSrjr<r=rr r rrrlogitsrBrC total_loss ignored_indexloss_fct start_lossend_losss r7rEz$MarkupLMForQuestionAnswering.forwards`&1%>> from transformers import AutoProcessor, AutoModelForTokenClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> processor.parse_html = False >>> model = AutoModelForTokenClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> nodes = ["hello", "world"] >>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"] >>> node_labels = [1, 2] >>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```NTr?rr9rArHrr) r4r0rrSrrr:rrr)r3rsr?r@rrrSrjrUrr r rrrrrArKs r7rEz&MarkupLMForTokenClassification.forwards\&1% 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoProcessor, AutoModelForSequenceClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```NTr?r regressionsingle_label_classificationmulti_label_classificationr9rW)r4r0rr'rS problem_typer:rer=rhrprrErrrrrr)r3rsr?r@rrrSrjrUrr r rrrrHrArKs r7rEz)MarkupLMForSequenceClassification.forwardhsZ&1%rpsD$ AA&!9G6>>2   H %/ bii/ dpph   299  RYY  RYY  bii$ ryy "!"))!$% II% <<% % << % LL4' %  %%05BII5r 2#.#N* bii* Z io i ir +r r ji #:i i X ] %<]  ] @ n (?n  n b rF