iuRdZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z dd lmZdd lmZmZmZmZmZmZdd lmZmZdd lmZdd lmZmZmZddl m!Z!ejDe#Z$ejJZ&GddejNZ( d@dejNdejRdejRdejRdejRdzde*de*fdZ+GddejNZ,GddejNZ-GddejNZ.Gd d!ejNZ/Gd"d#ejNZ0Gd$d%eZ1Gd&d'ejNZ2Gd(d)ejNZ3Gd*d+ejNZ4Gd,d-ejNZ5Gd.d/ejNZ6eGd0d1eZ7eGd2d3e7Z8eGd4d5e7Z9ed67Gd8d9e7Z:ed:7Gd;de7Z J ! "& '( ((  ( (  ( $  $$ $$ $ ^^J/ \\*- ) bYZ`a a bs,A,F77 G G  G)NNNNN)__name__ __module__ __qualname____doc__r%rZ __classcell__rAs@rBrr-s!Q &  5rCrmodulequerykeyvalueattention_maskscalingr9c tj||jdd|z}|#|ddddddd|jdf} || z}tj j |dtjj|j}tj j|||j}tj||} | jddj} | |fS)NrHrr")dimrF)ptrainingr) r;matmul transposeshaper functionalsoftmaxfloat32torFr9rk contiguous) rarbrcrdrerfr9kwargs attn_weights causal_mask attn_outputs rBeager_attention_forwardrxzs<<s}}Q':;gEL!$Q1o " o%=> #k1 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$rCc eZdZfdZ ddej dej dzdedzdeej fdZ xZ S) LayoutLMSelfAttentionc$t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ tj|j|j|_ tj|j|j|_ tj|j|j|_tj|j |_|j |_|jdz|_y)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()g)r$r%r(num_attention_headshasattr ValueErrorr@intattention_head_size all_head_sizerLinearrbrcrdr7attention_probs_dropout_probr9attention_dropoutrfr>s rBr%zLayoutLMSelfAttention.__init__sC     : : :a ?PVXhHi#F$6$6#78 445Q8   #)#=#= #&v'9'9Fs rBr%zLayoutLMSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rCr input_tensorrcr|j|}|j|}|j||z}|SNrr9r6r?rrs rBrZzLayoutLMSelfOutput.forward7 =1  ]3 }|'CD rCr[r\r]r%r;rrZr_r`s@rBrr1> U\\RWR^R^rCrc eZdZfdZ ddej dej dzdedzdeej fdZ xZ S) LayoutLMAttentioncbt|t||_t ||_yr)r$r%rzr?routputr>s rBr%zLayoutLMAttention.__init__s& )&1 (0 rCNrrerrc n|j|f||d|}|j|d|}|f|ddz}|SN)rerrr)r?r)r?rrerrt self_outputsattention_outputrs rBrZzLayoutLMAttention.forwards\!tyy  )/    ;;|A F#% QR(88rCrrr`s@rBrrsW148). ||))D0 $;  u||  rCrcVeZdZfdZdej dej fdZxZS)LayoutLMIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r$r%rrr(intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr>s rBr%zLayoutLMIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rCrrcJ|j|}|j|}|Sr)rrr?rs rBrZzLayoutLMIntermediate.forwards& =1 00? rCrr`s@rBrrs#9U\\ellrCrcneZdZfdZdej dej dej fdZxZS)LayoutLMOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r$r%rrrr(rr6r5r7r8r9r>s rBr%zLayoutLMOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rCrrrcr|j|}|j|}|j||z}|Srrrs rBrZzLayoutLMOutput.forward rrCrr`s@rBrrrrCrc eZdZfdZ d dej dej dzdedzdeej fdZ dZ xZ S) LayoutLMLayerct||j|_d|_t ||_t ||_t||_ y)Nr) r$r%chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr>s rBr%zLayoutLMLayer.__init__sI '-'E'E$*6208$V, rCNrrerrc |j|f||d|}|d}|dd}t|j|j|j|}|f|z}|Sr)rrfeed_forward_chunkrr) r?rrerrtself_attention_outputsrr layer_outputs rBrZzLayoutLMLayer.forwards"0 " )/"  "  2!4(,0  # #T%A%A4CSCSUe  /G+rCcL|j|}|j||}|Sr)rr)r?rintermediate_outputrs rBrz LayoutLMLayer.feed_forward_chunk2s,"//0@A{{#68HI rCr) r[r\r]r%r;rrrrrZrr_r`s@rBrrs\-48). ||))D0 $;  u||  .rCrceZdZfdZe d dej dejdzdedzdedzdedzde ej e zf d Z xZ S) LayoutLMEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) r$r%r@r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r?r@irAs rBr%zLayoutLMEncoder.__init__:sN  ]]5IaIaCb#caM&$9#cd &+#$dsA#Nrreroutput_hidden_states return_dictrc |rdnd}|rdnd}t|jD])\} } |r||fz}| |||fi|} | d}|s!|| dfz}+|r||fz}t|||S)Nrr)last_hidden_stater attentions) enumeraterr ) r?rrerrrrtall_hidden_statesall_self_attentionsr layer_module layer_outputss rBrZzLayoutLMEncoder.forward@s#7BD$5b4(4 POA|#$58H$H!(! M*!,M &9]1=Ms rBr%zLayoutLMPooler.__init__hs9 YYv1163E3EF '')rCrrc\|dddf}|j|}|j|}|S)Nr)rr)r?rfirst_token_tensor pooled_outputs rBrZzLayoutLMPooler.forwardms6+1a40 #56  6 rCrr`s@rBrrgs#$ U\\ellrCrcVeZdZfdZdej dej fdZxZS)LayoutLMPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r$r%rrr(rrrrr transform_act_fnr6r5r>s rBr%z(LayoutLMPredictionHeadTransform.__init__xs{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrCrrcl|j|}|j|}|j|}|Sr)rrr6rs rBrZz'LayoutLMPredictionHeadTransform.forwards4 =1 --m< }5 rCrr`s@rBrrws$UU\\ellrCrc$eZdZfdZdZxZS)LayoutLMLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) r$r%r transformrrr(r'decoder Parameterr;rJrr>s rBr%z!LayoutLMLMPredictionHead.__init__s[ 8@yy!3!3V5F5FTR LLV->->!?@ rCcJ|j|}|j|}|Sr)rrrs rBrZz LayoutLMLMPredictionHead.forwards$}5  ]3 rC)r[r\r]r%rZr_r`s@rBrrsArCrcVeZdZfdZdej dej fdZxZS)LayoutLMOnlyMLMHeadcBt|t||_yr)r$r%r predictionsr>s rBr%zLayoutLMOnlyMLMHead.__init__s 3F;rCsequence_outputrc(|j|}|Sr)r)r?rprediction_scoress rBrZzLayoutLMOnlyMLMHead.forwards ,,_=  rCrr`s@rBrrs#<!u||! !rCrcZeZdZUeed<dZdZejfdZ xZ S)LayoutLMPreTrainedModelr@layoutlmTcXt||t|tr t j |j yt|trZt j|jtj|jjdjdyy)zInitialize the weightsr"r!N)r$ _init_weightsrrinitzeros_rrcopy_r r;r<rnr=)r?rarAs rBrz%LayoutLMPreTrainedModel._init_weightssx f% f6 7 KK $  2 3 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h4rC) r[r\r]r__annotations__base_model_prefixsupports_gradient_checkpointingr;no_gradrr_r`s@rBrrs1 "&*#U]]_iirCrc6eZdZfdZdZdZee ddejdzdejdzdejdzdejdzd ejdzd ejdzd e dzd e dzd e dzde e zfdZxZS) LayoutLMModelct||||_t||_t ||_t||_|jyr) r$r%r@rrYrencoderrpooler post_initr>s rBr%zLayoutLMModel.__init__sG   ,V4&v. $V,  rCc.|jjSrrYr*r?s rBget_input_embeddingsz"LayoutLMModel.get_input_embeddingss...rCc&||j_yrr)r?rds rBset_input_embeddingsz"LayoutLMModel.set_input_embeddingss*/'rCNrMrNrerOr rPrrrrc  ||n|jj}||n|jj}| | n|jj} | | t d|#|j |||j } n!||j dd} n t d| |jn |j} |tj| | }|&tj| tj| }|)tj| dztj| }|jdjd } | j|j } d | z tj|jj z} |j#||||| }|j%|| ||d }|d}|j'|}t)|||j*|j,S)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, LayoutLMModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased") >>> model = LayoutLMModel.from_pretrained("microsoft/layoutlm-base-uncased") >>> words = ["Hello", "world"] >>> normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782] >>> token_boxes = [] >>> for word, box in zip(words, normalized_word_boxes): ... word_tokens = tokenizer.tokenize(word) ... token_boxes.extend([box] * len(word_tokens)) >>> # add bounding boxes of cls + sep tokens >>> token_boxes = [[0, 0, 0, 0]] + token_boxes + [[1000, 1000, 1000, 1000]] >>> encoding = tokenizer(" ".join(words), return_tensors="pt") >>> input_ids = encoding["input_ids"] >>> attention_mask = encoding["attention_mask"] >>> token_type_ids = encoding["token_type_ids"] >>> bbox = torch.tensor([token_boxes]) >>> outputs = model( ... input_ids=input_ids, bbox=bbox, attention_mask=attention_mask, token_type_ids=token_type_ids ... ) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer"z5You have to specify either input_ids or inputs_embeds)rGrE)rrH)rFg?)rMrNr rOrPT)rrrr)r pooler_outputrr)r@rruse_return_dictr%warn_if_padding_and_no_attention_maskrIrGr;onesrJrK unsqueezerrrFfinfominrYr r r rr)r?rMrNrerOr rPrrrrtrQrGextended_attention_maskembedding_outputencoder_outputsrrs rBrZzLayoutLMModel.forwardsj2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U%.%:!! @T@T  !"ZZ FCN  !"[[EJJvVN <;;{T1FSD"0":":1"="G"G"J"9"<"<4::"<"N#&)@#@EKKPTPZPZD[D_D_"_??%)' + ,,  #/!5 ' *!, O4 )-')77&11   rC) NNNNNNNNN)r[r\r]r%rrrrr; LongTensorrrrr rZr_r`s@rBr r s  /0.2(,37260426)-,0#'h ##d*h %h ))D0 h ((4/ h &&- h ((4/h  $;h #Tkh D[h  + +h h rCr cfeZdZdddZfdZdZdZdZee dd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde dzde dzde dzdeezfdZxZS)LayoutLMForMaskedLMzcls.predictions.biasz*layoutlm.embeddings.word_embeddings.weight)zcls.predictions.decoder.biaszcls.predictions.decoder.weightct||t||_t ||_|j yr)r$r%r rrclsr r>s rBr%zLayoutLMForMaskedLM.__init__;s4  %f- &v. rCcB|jjjSrrrYr*rs rBrz(LayoutLMForMaskedLM.get_input_embeddingsD}}''777rCcB|jjjSr)r$rrrs rBget_output_embeddingsz)LayoutLMForMaskedLM.get_output_embeddingsGsxx##+++rCc||jj_|j|jj_yr)r$rrr)r?new_embeddingss rBset_output_embeddingsz)LayoutLMForMaskedLM.set_output_embeddingsJs,'5$$2$7$7!rCNrMrNrerOr rPlabelsrrrrc p| | n|jj} |j|||||||| d } | d} |j| }d}|Ft }||j d|jj |j d}t||| j| jS)a2 bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, LayoutLMForMaskedLM >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased") >>> model = LayoutLMForMaskedLM.from_pretrained("microsoft/layoutlm-base-uncased") >>> words = ["Hello", "[MASK]"] >>> normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782] >>> token_boxes = [] >>> for word, box in zip(words, normalized_word_boxes): ... word_tokens = tokenizer.tokenize(word) ... token_boxes.extend([box] * len(word_tokens)) >>> # add bounding boxes of cls + sep tokens >>> token_boxes = [[0, 0, 0, 0]] + token_boxes + [[1000, 1000, 1000, 1000]] >>> encoding = tokenizer(" ".join(words), return_tensors="pt") >>> input_ids = encoding["input_ids"] >>> attention_mask = encoding["attention_mask"] >>> token_type_ids = encoding["token_type_ids"] >>> bbox = torch.tensor([token_boxes]) >>> labels = tokenizer("Hello world", return_tensors="pt")["input_ids"] >>> outputs = model( ... input_ids=input_ids, ... bbox=bbox, ... attention_mask=attention_mask, ... token_type_ids=token_type_ids, ... labels=labels, ... ) >>> loss = outputs.loss ```NT)rerOr rPrrrrr"losslogitsrr) r@rrr$rrr'rrr)r?rMrNrerOr rPr-rrrrtrrrmasked_lm_lossloss_fcts rBrZzLayoutLMForMaskedLM.forwardNs@&1%s rBr%z*LayoutLMForSequenceClassification.__init__i   ++%f- zz&"<"<= ))F$6$68I8IJ rCcB|jjjSrr&rs rBrz6LayoutLMForSequenceClassification.get_input_embeddingsr'rCNrMrNrerOr rPr-rrrrc | | n|jj} |j|||||||| d } | d} |j| } |j | }d}||jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|j dk(r&||j|j}n|||}n|jj dk(r=t}||jd|j |jd}n,|jj dk(rt}|||}t!||| j"| j$ S) aB bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, LayoutLMForSequenceClassification >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased") >>> model = LayoutLMForSequenceClassification.from_pretrained("microsoft/layoutlm-base-uncased") >>> words = ["Hello", "world"] >>> normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782] >>> token_boxes = [] >>> for word, box in zip(words, normalized_word_boxes): ... word_tokens = tokenizer.tokenize(word) ... token_boxes.extend([box] * len(word_tokens)) >>> # add bounding boxes of cls + sep tokens >>> token_boxes = [[0, 0, 0, 0]] + token_boxes + [[1000, 1000, 1000, 1000]] >>> encoding = tokenizer(" ".join(words), return_tensors="pt") >>> input_ids = encoding["input_ids"] >>> attention_mask = encoding["attention_mask"] >>> token_type_ids = encoding["token_type_ids"] >>> bbox = torch.tensor([token_boxes]) >>> sequence_label = torch.tensor([1]) >>> outputs = model( ... input_ids=input_ids, ... bbox=bbox, ... attention_mask=attention_mask, ... token_type_ids=token_type_ids, ... labels=sequence_label, ... ) >>> loss = outputs.loss >>> logits = outputs.logits ```NT rMrNrerOr rPrrrr regressionsingle_label_classificationmulti_label_classificationr"r/)r@rrr9r< problem_typer;rFr;rKrrsqueezerrrrrr)r?rMrNrerOr rPr-rrrrtrrr1r0r3s rBrZz)LayoutLMForSequenceClassification.forwards@&1%s rBr%z'LayoutLMForTokenClassification.__init__=r=rCcB|jjjSrr&rs rBrz3LayoutLMForTokenClassification.get_input_embeddingsGr'rCNrMrNrerOr rPr-rrrrc ~| | n|jj} |j|||||||| d } | d} |j| } |j | }d}|>> from transformers import AutoTokenizer, LayoutLMForTokenClassification >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased") >>> model = LayoutLMForTokenClassification.from_pretrained("microsoft/layoutlm-base-uncased") >>> words = ["Hello", "world"] >>> normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782] >>> token_boxes = [] >>> for word, box in zip(words, normalized_word_boxes): ... word_tokens = tokenizer.tokenize(word) ... token_boxes.extend([box] * len(word_tokens)) >>> # add bounding boxes of cls + sep tokens >>> token_boxes = [[0, 0, 0, 0]] + token_boxes + [[1000, 1000, 1000, 1000]] >>> encoding = tokenizer(" ".join(words), return_tensors="pt") >>> input_ids = encoding["input_ids"] >>> attention_mask = encoding["attention_mask"] >>> token_type_ids = encoding["token_type_ids"] >>> bbox = torch.tensor([token_boxes]) >>> token_labels = torch.tensor([1, 1, 0, 0]).unsqueeze(0) # batch size of 1 >>> outputs = model( ... input_ids=input_ids, ... bbox=bbox, ... attention_mask=attention_mask, ... token_type_ids=token_type_ids, ... labels=token_labels, ... ) >>> loss = outputs.loss >>> logits = outputs.logits ```NTr@rr"r/) r@rrr9r<rrr;rrr)r?rMrNrerOr rPr-rrrrtrrr1r0r3s rBrZz&LayoutLMForTokenClassification.forwardJs|&1%>> from transformers import AutoTokenizer, LayoutLMForQuestionAnswering >>> from datasets import load_dataset >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("impira/layoutlm-document-qa", add_prefix_space=True) >>> model = LayoutLMForQuestionAnswering.from_pretrained("impira/layoutlm-document-qa", revision="1e3ebac") >>> dataset = load_dataset("nielsr/funsd", split="train") >>> example = dataset[0] >>> question = "what's his name?" >>> words = example["words"] >>> boxes = example["bboxes"] >>> encoding = tokenizer( ... question.split(), words, is_split_into_words=True, return_token_type_ids=True, return_tensors="pt" ... ) >>> bbox = [] >>> for i, s, w in zip(encoding.input_ids[0], encoding.sequence_ids(0), encoding.word_ids(0)): ... if s == 1: ... bbox.append(boxes[w]) ... elif i == tokenizer.sep_token_id: ... bbox.append([1000] * 4) ... else: ... bbox.append([0] * 4) >>> encoding["bbox"] = torch.tensor([bbox]) >>> word_ids = encoding.word_ids(0) >>> outputs = model(**encoding) >>> loss = outputs.loss >>> start_scores = outputs.start_logits >>> end_scores = outputs.end_logits >>> start, end = word_ids[start_scores.argmax(-1)], word_ids[end_scores.argmax(-1)] >>> print(" ".join(words[start : end + 1])) M. Hamann P. Harper, P. Martinez ```NTr@rrr")ri) ignore_indexrH)r0 start_logits end_logitsrr)r@rrrNsplitrErslenrIclamprrrr)r?rMrNrerOr rPrQrRrrrrtrrr1rUrV total_loss ignored_indexr3 start_lossend_losss rBrZz$LayoutLMForQuestionAnswering.forwardsD&1%r^collections.abcrr;rtorch.nnrrrr r activationsr modeling_layersr modeling_outputsr r rrrrmodeling_utilsrr pytorch_utilsrutilsrrrconfiguration_layoutlmr get_loggerr[loggerr6r4Modulerrfloatrxrzrrrrrrrrrrrr r"r8rGrL__all__rrCrBrms_$ AA&!9G6>>2   H %LLIIh% II% <<% % << % LL4' %  %%05BII5r   2 299  RYY #.#N* bii* \ RYY  bii$ ryy "!"))! io i i| +| | ~w 1w w t } (?}  } @i %<i i X@ #:@ @ F rC