ixVdZddlZddlmZddlmZddlmZddlZ ddl Z ddl m Z ddl m Z dd lmZdd lmZdd lmZmZdd lmZmZdd lmZddlmZmZmZmZddlm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ejNe(Z)eedGddeZ*eedGddeZ+dZ,Gdde jZZ.Gdde jZZ/ dCde jZd e j`d!e j`d"e j`d#e j`dzd$e1dzd%e1d&eefd'Z2Gd(d)e jZZ3Gd*d+e jZZ4Gd,d-e jZZ5Gd.d/e jZZ6Gd0d1e jZZ7Gd2d3eZ8Gd4d5e jZZ9eGd6d7eZ:eGd8d9e:Z;Gd:d;e jZZ<ed<Gd=d>e:Z=ed?Gd@dAe:Z>gdBZ?y)Dz,PyTorch VideoMAE (masked autoencoder) model.N)Callable)deepcopy) dataclass)nn)MSELoss)ACT2FN)GradientCheckpointingLayer)BaseModelOutputImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)can_return_tuplecheck_model_inputs)VideoMAEConfigz[ Class for VideoMAEDecoder's outputs, with potential hidden states and attentions. ) custom_introceZdZUdZdZej dzed<dZe ej dzed<dZ e ej dzed<y)VideoMAEDecoderOutputz logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. Nlogits hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rtuplerx/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/videomae/modeling_videomae.pyrr(sR (,FE   $+59M5**+d2926Je''(4/6r)rzb Class for VideoMAEForPreTraining's outputs, with potential hidden states and attentions. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dzed<dZ e ej dzed<y)VideoMAEForPreTrainingOutputz loss (`torch.FloatTensor` of shape `(1,)`): Pixel reconstruction loss. logits (`torch.FloatTensor` of shape `(batch_size, patch_size ** 2 * num_channels)`): Pixel reconstruction logits. Nlossrrr) r r!r"r#r-r$r%r&rrr'rr(r)r*r,r,9sg&*D%  d ")'+FE   $+59M5**+d2926Je''(4/6r)r,chfd}tjt|Dcgc] }|| c}}tj|dddddf|dddddf<tj|dddddf|dddddf<t j |jdScc}w)z Sinusoid position encoding tablec tDcgc]$}|tjdd|dzzz z &c}Scc}w)Ni')rangenppower)positionhid_jd_hids r*get_position_angle_vecz;get_sinusoid_encoding_table..get_position_angle_vecSs;RWX]R^_288E1 +;e+CDD___s);Nrr0r)r2arrayr1sincosr$r% unsqueeze) n_positionr6r7pos_isinusoid_tables ` r*get_sinusoid_encoding_tabler?Os`XX%PZJ[\5e<\]N ff^Aqt!tG%<=N1add7 ff^Aqt!tG%<=N1add7   ^ , 6 6q 99 ]sB/c(eZdZdZfdZdZxZS)VideoMAEEmbeddingsz7 Construct the patch and position embeddings. ct|t||_|jj|_t |j|j |_||_yN) super__init__VideoMAEPatchEmbeddingspatch_embeddings num_patchesr? hidden_sizeposition_embeddingsconfigselfrK __class__s r*rEzVideoMAEEmbeddings.__init__csR  7 ?00<<#>t?O?OQWQcQc#d  r)c |j|}||jjj|j |j dz}|)|j \}}}||}|j|d|}|S)NTdevicecopy)rGrJdetachtype_astorQshapereshape)rM pixel_valuesbool_masked_pos embeddings batch_size_ num_channelss r*forwardzVideoMAEEmbeddings.forwardls**<8  $":":"A"A"C"K"KJ"W"Z"Z$$4#[#   &*4*:*: 'J<#_$45J#++JLIJr)r r!r"r#rEr_ __classcell__rNs@r*rArA]s r)rAc(eZdZdZfdZdZxZS)rFaw Video to Patch Embedding. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size) * (height // patch_size) * (width // patch_size). c t ||j}|j}|j}|j }|j }|j}t|tjjr|n||f}t|tjjr|n||f}||_||_t||_|d|dz|d|dzz||jzz}||_||_ tj|||j|d|df|j|d|df|_y)Nrr) in_channels out_channels kernel_sizestride)rDrE image_size patch_sizer^rI num_frames tubelet_size isinstance collectionsabcIterableintrHrConv3d projection) rMrKrirjr^rIrkrlrHrNs r*rEz VideoMAEPatchEmbeddings.__init__s> && && ** (( && ** #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq $$ - ]jm + 1 A0N OS]aeararSr s )&))$$**JqM:a=I%%z!}jmD  r)c |j\}}}}}||jk7r td||jdk7s||jdk7r2td|d|d|jdd|jdd |j dddd d }|j |j djdd}|S) NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (*z) doesn't match model (z).r0r)rWr^ ValueErrorripermutersflatten transpose)rMrYr\rkr^heightwidthr[s r*r_zVideoMAEPatchEmbeddings.forwards>J>P>P; J fe 4,, ,w  T__Q' '5DOOA4F+F$VHAeW4KDOO\]L^K__`aeapapqras`ttvw $++Aq!Q: __\2::1=GG1M r)r`rbs@r*rFrF~s 6 r)rFmodulequerykeyvalueattention_maskscalingdropoutkwargsc ||jddz}tj||jdd|z}|#|ddddddd|jdf}||z}t j j|d}t j j|||j}tj||} | jddj} | |fS) NrSr0rdim)ptrainingr) sizer$matmulrzrWr functionalsoftmaxrr contiguous) r}r~rrrrrr attn_weights attn_outputs r*eager_attention_forwardrs**R.D(<<s}}Q':;gEL!'1a399R=(@A#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r)ceZdZdeddffd Zddej dzdeej ej ffdZxZ S)VideoMAESelfAttentionrKreturnNct||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |j|_ |jdz|_ d|_ tj|j|jd|_tj|j|jd|_tj|j|jd|_|j&rotj(t+j,|j|_tj(t+j,|j|_yd|_d|_y) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .rFbias)rDrErInum_attention_headshasattrrwrKrqattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearr~rrqkv_bias Parameterr$zerosq_biasv_biasrLs r*rEzVideoMAESelfAttention.__init__s     : : :a ?PVXhHi"6#5#5"67334A7  #)#=#= #&v'9'9FS"S%--.EF o--r)rC) r r!r"rrEr$Tensorr'r_rarbs@r*rrsG~$4.U\\D%8.E%,,X]XdXdJdDe.r)rcxeZdZdZdeffd ZdejdejdejfdZxZ S)VideoMAESelfOutputz The residual connection is defined in VideoMAELayer instead of here (as is the case with other models), due to the layernorm applied before each block. rKct|tj|j|j|_tj |j|_yrC) rDrErrrIdenseDropouthidden_dropout_probrrLs r*rEzVideoMAESelfOutput.__init__sB YYv1163E3EF zz&"<"<= r)r input_tensorrcJ|j|}|j|}|SrCrrrMrrs r*r_zVideoMAESelfOutput.forwards$ =1  ]3 r) r r!r"r#rrEr$rr_rarbs@r*rrs= >~> U\\RWR^R^r)rc\eZdZdeffd Zdej dej fdZxZS)VideoMAEAttentionrKcbt|t||_t ||_yrC)rDrEr attentionroutputrLs r*rEzVideoMAEAttention.__init__!s& .v6(0 r)rrcR|j|\}}|j||}|SrC)rr)rMrself_attn_outputr]rs r*r_zVideoMAEAttention.forward&s,"nn];!-}= r) r r!r"rrEr$rr_rarbs@r*rr s*1~1 U\\ellr)rc\eZdZdeffd Zdej dej fdZxZS)VideoMAEIntermediaterKct|tj|j|j |_t|jtrt|j|_ y|j|_ yrC) rDrErrrIintermediate_sizerrm hidden_actstrr intermediate_act_fnrLs r*rEzVideoMAEIntermediate.__init__.s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r)rrcJ|j|}|j|}|SrC)rr)rMrs r*r_zVideoMAEIntermediate.forward6s& =1 00? r)rrbs@r*rr-s*9~9U\\ellr)rcteZdZdeffd Zdej dej dej fdZxZS)VideoMAEOutputrKct|tj|j|j |_tj|j|_ yrC) rDrErrrrIrrrrrLs r*rEzVideoMAEOutput.__init__>sB YYv779K9KL zz&"<"<= r)rrrcT|j|}|j|}||z}|SrCrrs r*r_zVideoMAEOutput.forwardCs. =1  ]3 % 4 r)rrbs@r*rr=s8>~> U\\RWR^R^r)rc`eZdZdZdeffd ZdejdejfdZxZ S) VideoMAELayerz?This corresponds to the Block class in the timm implementation.rKcrt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nreps)rDrEchunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormrIlayer_norm_epslayernorm_beforelayernorm_afterrLs r*rEzVideoMAELayer.__init__Ns '-'E'E$*6208$V, " V-?-?VEZEZ [!||F,>,>FDYDYZr)rrc|j|}|j|}||z}|j|}|j|}|j ||}|SrC)rrrrr)rMrhidden_states_normattention_output layer_outputs r*r_zVideoMAELayer.forwardXsi!22=A>>*<=)=8 ++M: ((6 {{<? r)rrbs@r*rrKs/I[~[U\\ellr)rcHeZdZdeffd Zdej defdZxZ S)VideoMAEEncoderrKct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) rDrErKr ModuleListr1num_hidden_layersrlayergradient_checkpointing)rMrKr]rNs r*rEzVideoMAEEncoder.__init__ksN  ]]5IaIaCb#caM&$9#cd &+#$dsA#rrcdt|jD] \}}||}t|S)Nlast_hidden_state) enumeraterr )rMri layer_modules r*r_zVideoMAEEncoder.forwardqs5(4 8OA|(7M 8??r)) r r!r"rrEr$rr r_rarbs@r*rrjs),~, @U\\@o@r)rcJeZdZUeed<dZdZdZdZddgZ dZ dZ dZ dZ eedZy ) VideoMAEPreTrainedModelrKvideomaerYvideoTrAr)rrN)r r!r"rr&base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsr(r)r*rrxsO "$O&*#-?N"&&+r)rc eZdZfdZdZede d dejdejdzde e d e fd Z xZS) VideoMAEModelct||||_t||_t ||_|jrd|_n0tj|j|j|_|jy)Nr)rDrErKrAr[rencoderuse_mean_pooling layernormrrrIr post_initrLs r*rEzVideoMAEModel.__init__si   ,V4&v.  " "!DN\\&*<*<&BWBWXDN r)c.|jjSrC)r[rG)rMs r*get_input_embeddingsz"VideoMAEModel.get_input_embeddingss///r)F)tie_last_hidden_statesNrYrZrrc |j||}|j|}|j}|j|j|}t |S)aB bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. If `None`, then all patches are considered. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> import torch >>> from transformers import VideoMAEVideoProcessor, VideoMAEModel >>> from huggingface_hub import hf_hub_download >>> # replace this with your own video file >>> video_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> video_processor = VideoMAEVideoProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEModel.from_pretrained("MCG-NJU/videomae-base") >>> # prepare video for the model >>> inputs = video_processor(video_path, return_tensors="pt") >>> # forward pass >>> with torch.no_grad(): ... outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 1568, 768] ```r)r[r rrr )rMrYrZrembedding_outputencoder_outputssequence_outputs r*r_zVideoMAEModel.forwardsSR ??<I+/<<8H+I);; >> %"nn_=OAAr)rC)r r!r"rErrrr$r% BoolTensorrrr r_rarbs@r*r r sv 0u548.B''.B))D0.B+, .B  .B6.Br)r cHeZdZdeffd Zdej defdZxZ S)VideoMAEDecoderrKct||j|jz|jdzz}t |}|j |_|j|_ |j|_ |j|_ tjt!|jDcgc] }t#|c}|_tj&|j |_|dkDr tj*|j |ntj,|_d|_||_ycc}w)Nr0rF)rDrEr^rlrjrdecoder_hidden_sizerIdecoder_num_hidden_layersrdecoder_num_attention_headsrdecoder_intermediate_sizerrrr1rdecoder_layersrnormrIdentityheadrrK)rMrKdecoder_num_labelsdecoder_configr]rNs r*rEzVideoMAEDecoder.__init__s #0063F3FFIZIZ\]I]]!&)%+%?%?"+1+K+K(-3-O-O*+1+K+K( mm49&:Z:Z4[ \q]> * \ LL!;!;< I[^_I_BIIf002D Eegepeper ',#$  ]s.D=rreturn_token_numc|jD] }||} |dd| df}|j|}|j|}t|S)N)r)r r!r#r)rMrr&rrs r*r_zVideoMAEDecoder.forwardsb // 8L(7M 8&a*:):);&;<  -0 =)$F33r)) r r!r"rrEr$rrqr_rarbs@r*rrs&%~%, 4U\\ 4S 4r)rzb The VideoMAE Model transformer with the decoder on top for self-supervised pre-training. c xeZdZfdZeedejdejde e de fdZ xZ S)VideoMAEForPreTrainingct||||_t||_t j |j|jd|_ t jtjdd|j|_ t|jjj |j|_t%||_|j)y)NFrr)rDrErKr rrrrIrencoder_to_decoderrr$r mask_tokenr?r[rHrJrdecoderrrLs r*rEzVideoMAEForPreTraining.__init__s   %f- "$))F,>,>@Z@Zaf"g,,u{{1a9S9S'TU#> MM $ $ 0 0&2L2L$  'v.  r)rYrZrrc *|j|fd|i|}|j}|j|}|j\}}}| t d|j j |ddj|} | jj|jd} | |j|d|} | |j|d|} tj|| z|j| zgd} |j| | jd} | j }d}tj"5|j$j&d k7r|}n|j}|j(}tj*t,j|| ddddddf}tj*t.j|| ddddddf}||z|z}|j\}}}}}|j$j0|j$j2}}|j$j4r|j7|||z||||z|||z|}|j9d dd d dddd j;}|j7|||z|z|z|z|z||z|z|}||j=ddz |j?dddjAdzz }|j7|||z|z|z|z|z||z|z|z}n|j$j&d k7r t d|j7|||z||||z|||z|}|j9d dd d dddd j;}|j7|||z|z|z|z|z||z|z|z}|j\}}}||j|d|}dddtC}||}tE|||jF|jHS#1swY?xYw)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Each video in the batch must have the same number of masked patches. Sequence length is `(num_frames // tubelet_size) * (image_size // patch_size) ** 2`. Examples: ```python >>> from transformers import AutoImageProcessor, VideoMAEForPreTraining >>> import numpy as np >>> import torch >>> num_frames = 16 >>> video = list(np.random.randint(0, 256, (num_frames, 3, 224, 224))) >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base") >>> model = VideoMAEForPreTraining.from_pretrained("MCG-NJU/videomae-base") >>> pixel_values = image_processor(video, return_tensors="pt").pixel_values >>> num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2 >>> seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame >>> bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss = outputs.loss ```rZNz!One must provided a boolean mask rSTrPrrr)rQdtyperrvr0r)rkeepdim)runbiasedr3gư>zQCan't unnormalize non-RGB images. Consider setting config.norm_pix_loss to False.r-rrr)%rrr+rWrwrJexpandrUrTrVrQrXr$catr,r-rno_gradrKr^r/ as_tensorrrrlrj norm_pix_lossrrxrmeanvarsqrtrr,rr)rMrYrZroutputsrr\r]r^expanded_position_embeddingspos_emb_visible pos_emb_maskx_fulldecoder_outputsrr-framesrQr/r;stdtimer{r|rlrj frames_norm videos_patchlabelsloss_fcts r*r_zVideoMAEForPreTraining.forward sF$14==#i#ibh#i!3311/B'6&;&;# A|  "@A A'+'?'?'F'FzSUWY'Z'b'bco'p$'C'J'J'L'O'OWcWjWjqu'O'v$67GHPPQ[]_amn3ODLLZY[]ij Oo=tQ]?]^def26flFXFXYZF[1\ '' ]]_H Y{{''1,%&,,$**'<=@@V[@\]acgijlprv]vwoo&:;>>fTY>Z[_aeghjnpt[tu%+d2J P  P)r r!r"rErrr$r%rrrr,r_rarbs@r*r)r)sb "L ''L ))L +, L & L L r)r)z VideoMAE Model transformer with a video classification head on top (a linear layer on top of the average pooled hidden states of all tokens) e.g. for ImageNet. c eZdZfdZee ddejdzdejdzdee de fdZ xZ S) VideoMAEForVideoClassificationct|||j|_t||_|j rt j|jnd|_ |jdkDr*t j|j|jnt j|_ |jy)Nr)rDrE num_labelsr rrrrrIfc_normrr" classifierrrLs r*rEz'VideoMAEForVideoClassification.__init__s   ++%f-  1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import torch >>> from transformers import VideoMAEVideoProcessor, VideoMAEForVideoClassification >>> from huggingface_hub import hf_hub_download >>> # replace this with your own video file >>> video_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> video_processor = VideoMAEVideoProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics") >>> inputs = video_processor(video_path, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) eating spaghetti ```Nrrr5) rrrOr;rP loss_functionrKr rr) rMrYrIrr>rrrr-s r*r_z&VideoMAEForVideoClassification.forwardsR$14==#H#H!33 << #$))!,F\\&)F$QT*F(  %4%%ffdkkLVLD$!//))   r))NN) r r!r"rErrr$rrrr r_rarbs@r*rLrLsj -1&*; llT);  t#; +, ;  ; ; r)rL)r)r rrL)Nr)@r#collections.abcrnrrRr dataclassesrnumpyr2r$rtorch.nnr activationsr modeling_layersr modeling_outputsr r modeling_utilsr rprocessing_utilsrutilsrrrrutils.constantsrr utils.genericrrconfiguration_videomaer get_loggerr loggerrr,r?ModulerArFrfloatrrrrrrrrrr rr)rL__all__r(r)r*resl3$! !9FF&MMJA2   H %  7K7  7  7; 7  7 :B2bii2x!% II% <<% % << % LL4' % T\ %%'(%:9.BII9.z$   299  RYY .> @bii @o"CB+CBCBL"4bii"4J ` 4`  ` F K %<K  K \ sr)