iddlZddlmZmZmZddlZddlmZddlm Z m Z ddl m Z ddl mZmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZGddej>Z Gddej>Z!Gddej>Z"Gddej>Z#Gddej>Z$Gddej>Z%Gddeee Z&dZ'y)N)OptionalTupleUnion) ConfigMixinregister_to_config)apply_forward_hook) Attention SpatialNorm) DecoderOutputDiagonalGaussianDistribution) Downsample2D)AutoencoderKLOutput) ModelMixin) ResnetBlock2D) Upsample2D)AutoencoderMixinceZdZdZ ddedeededededed ed dffd Ze d e jd e jfd Z d e jded e jfdZ xZS)AllegroTemporalConvLayera  Temporal convolutional layer that can be used for video (sequence of images) input. Code adapted from: https://github.com/modelscope/modelscope/blob/1509fdb973e5871f37148a4b5e5964cafd43e64d/modelscope/models/multi_modal/video_synthesis/unet_sd.py#L1016 Nin_dimout_dimdropoutnorm_num_groups up_sample down_samplestridereturnc Lt ||xs|}t|dz dzx}} d} ||_||_|rat j t j||t jt j||d||fdd|| f|_ n|rct j t j||t jt j||dzd||fd|| f|_ n_t j t j||t jt j||d||f| || f|_ t j t j||t jt j|t j||d||f| || f|_ t j t j||t jt j|t j||d||f| ||f|_ t j t j||t jt j||d||f| ||f|_y) Nrg?rr )r rr)rpaddingr!r)super__init__intrrnn Sequential GroupNormSiLUConv3dconv1Dropoutconv2conv3conv4) selfrrrrrrrpad_hpad_wpad_t __class__s ~/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/diffusers/models/autoencoders/autoencoder_kl_allegro.pyr$z!AllegroTemporalConvLayer.__init__(s( #VVaZ3.//&"  _f5  &'Avv+>y[\^cejZklDJ  _f5  &'A+66/BQPUW\L]^DJ  _f5  &'Avv+>PUW\H]^DJ ]] LL' 2 GGI JJw  IIgv66':UESXDY Z   ]] LL' 2 GGI JJw  IIgv66':UESXDY Z   ]] LL' 2 GGI IIgv66':UESXDY Z  hidden_statesctj|ddddddf|fd}tj||ddddddffd}|S)Nrrr )dim)torchcat)r7s r5_pad_temporal_dimz*AllegroTemporalConvLayer._pad_temporal_dim_sP =Aqs#;]"KQRS  =-1bc 2J"KQRS r6 batch_sizec.|jd|dfjddddd}|jr|dddddddf}n3|jr%|j dd|j ddz}n|}|js |jr|j |}n"|j|}|j |}|jr6|jddjdddddd jdd}|j|}|j|}|j|}|j|}|j|}|j|}||z}|jdddddjdd}|S) Nrr:r rr)r9 output_size)r r:) unflattenpermuterrrepeat_interleaveshaper+r=flattenr-r.r/)r0r7r>identitys r5forwardz AllegroTemporalConvLayer.forwardes%//J3CDLLQPQSTVWYZ[   $Q3Q3Y/H ^^$66qa]M`M`abMcfgMg6hH$H   t~~ JJ}5M 22=AM JJ}5M >>)33Aw?GG1aQRTUWXYaabcefgM..}=  =1 ..}=  =1 ..}=  =1  =0 %--aAq!<DDQJ r6)N FFr)__name__ __module__ __qualname____doc__r%rfloatboolr$ staticmethodr;Tensorr=rI __classcell__r4s@r5rr"s"&!!5 5 #5  5  5  5 5 5  5 n%,, U\\su||r6rceZdZ ddedededededededed ed ed ed ed effd ZdejdejfdZ xZ S)AllegroDownBlock3D in_channels out_channelsr num_layers resnet_epsresnet_time_scale_shift resnet_act_fn resnet_groupsresnet_pre_normoutput_scale_factorspatial_downsampletemporal_downsampledownsample_paddingct|g}g}t|D]M}|dk(r|n|}|jt ||d|||||| |  |jt ||d|Ot j||_t j||_ | rt ||d|dd|_ | |_ | r*t jt|d|| d g|_ yd|_ y) Nr rXrY temb_channelsepsgroupsrtime_embedding_norm non_linearityr`pre_norm皙?rrTr)rrrrop)use_convrYr!name)r#r$rangeappendrrr& ModuleListresnets temp_convstemp_convs_downadd_temp_downsampler downsamplers)r0rXrYrrZr[r\r]r^r_r`rarbrcrtruir4s r5r$zAllegroDownBlock3D.__init__s  z" A)*a+\K NN +!-"&"(#(?"/(;,     (  $1  ! 2}}W- -- 3 #;lCdhqr$D $7  "  $t,Xjqu!D !%D r6r7rc|jd}|jdddddjdd}t|j|j D]\}}||d}|||}|j r|j||}|j|jD] }||} |jd|dfjddddd}|S Nrr rrr@)temb)r>r:) rFrDrGziprtrurwrvrxrC)r0r7r>resnet temp_conv downsamplers r5rIzAllegroDownBlock3D.forwards"((+ %--aAq!<DDQJ !$T\\4??!C L FI"=tdefaultswishrKT?TFr rLrMrNr%rPstrrQr$r;rSrIrTrUs@r5rWrWs  '0$ $%(#'$)"#@%@%@% @%  @%  @%"%@%@%@%@%#@%!@%"@% @%DU\\ellr6rWceZdZ ddedededededededed ed ed ed ed eeffd Zde jde jfdZ xZ S)AllegroUpBlock3DrXrYrrZr[r\r]r^r_r`spatial_upsampletemporal_upsamplerfct|g}g}t|D]M}|dk(r|n|}|jt ||| |||||| |  |jt ||d|Ot j||_t j||_ | |_ | rt ||d|dd|_ | r(t jt|d|g|_ yd|_ y) NrrerlrmTr)rrrr)rorY)r#r$rqrrrrr&rsrtruadd_temp_upsample temp_conv_upr upsamplers)r0rXrYrrZr[r\r]r^r_r`rrrfrtruryinput_channelsr4s r5r$zAllegroUpBlock3D.__init__s  z" A,-F[ N NN .!-"/"(#(?"/(;,     (  $1  # 4}}W- -- 3!2  8lCbfop!D   mmZ tbn-o,pqDO"DOr6r7rc|jd}|jdddddjdd}t|j|j D]\}}||d}|||}|j r|j||}|j|jD] }||} |jd|dfjddddd}|Sr{) rFrDrGr}rtrurrrrC)r0r7r>r~r upsamplers r5rIzAllegroUpBlock3D.forwards"((+ %--aAq!<DDQJ !$T\\4??!C L FI"=tattnr~rs r5rIzAllegroMidBlock3DConv.forwards"((+ %--aAq!<DDQJ ' Q DA **=ZP '*4??DLLresidualrs r5rIzAllegroEncoder3D.forwards\\!_ 1aA.66q!<f%!!!j"%56>>q!Q1M""6*("  "t'B'B".. O :::vN O66t~~vNF#.. , #F+ ,^^F+F1aA.66q!<##F+v&!!!j"%56>>q!Q1M##F+("1aA.66q!<v&!!!j"%56>>q!Q1M r6 rLrMrNr%rrrQr$r;rSrIrTrUs@r5rrs- /C7Q !!Q,Q,Q, S/ Q,"#s(OQ,%*$)$4Q,Q,Q,Q,Q,f(ell(u||(r6rceZdZdddgdddddd f d ed ed eed fdeed fdeed fdedededeffd ZdejdejfdZ xZ S)AllegroDecoder3Dr@rrrrrFTTFrr rKrgrouprXrYup_block_types.temporal_upsample_blocksrrrr norm_typec t|tj||dddd|_tj |d|ddd|_d|_tjg|_ | dk(r|nd} t|dd |d| d k(rd n| |d|| |_tt|} | d } t|D]g\} }| }| | } | t|dz k(}|dk(rt|dz|| | || d ||| |  }n t!d|jj#|| }i| dk(rt%|d | |_n tj(|d |d |_tj*|_tj |d |d dd|_tj|d |dd|_d|_y)Nr:rrrrrr"rrrrrrr) rZrXrYrrr[r]r^rfr\z?Invalid `UP_block_type` encountered. Must be `AllegroUpBlock3D`rF)r#r$r&rrr*rrrs up_blocksrlistreversedrrrrrrr rr(r)rrrr)r0rXrYrrrrrrrrfreversed_block_out_channelsrry up_block_typeprev_output_channelrup_blockr4s r5r$zAllegroDecoder3D.__init__s " yy  r "   II&8&<>PQS>TV_irsr*'0I'= 4 /*2. !1:g1EI91"5)'  '+84F+G&H#4Q7 ). 9 1 A}"0 8;N#&8"9A"==N 22+/!3 3!/)7%7&>q&A#"("1"/,5 !!bcc NN ! !( +"0 / 14  !!,-?-BM!RD !#;Ma;P]lrv!wD   YY'9!'<>PQR>SU^hqr "4Q"7qRST &+#r6rrcv|jd}|jdddddjdd}|j|}|j d|dfjddddd}|}|j |}||z}t t|jjj}tjrL|jr@|j|j|}|jD]}|j||}n;|j|}|j!|}|jD] }||} |jdddddjdd}|j#|}|j%|}|j d|dfjddddd}|}|j'|}||z}|jdddddjdd}|j)|}|j d|dfjddddd}|Sr)rFrDrGrrCrnextiterr parametersdtyper;rrrrtorrrr)r0rr>r upscale_dtypers r5rIzAllegroDecoder3D.forwardvs'\\!_ 1aA.66q!<f%!!!j"%56>>q!Q1M""6*("T$..";";"=>?EE  "t'B'B66t~~vNF!NN M::8VL M ^^F+FYY}-F!NN *!&) *1aA.66q!<##F+v&!!!j"%56>>q!Q1M##F+("1aA.66q!<v&!!!j"%56>>q!Q1M r6rrUs@r5rrs+ 6P.B !! U,U,U,c3h U,#(c "2U,"#s(OU,U,U,U,U,n,ell,u||,r6rc"eZdZdZdZe d#dededeedfdeedfdeedfd ee dfd ee dfd ed ed edede dede de ddf fd Z de jde jfdZe d$de jde deeeeffdZde jde jfdZed$de jde deee jffdZde jde jfdZde jde jfdZ d%de jd e de d!ee j4deee jff d"ZxZS)&AutoencoderKLAllegroa! A VAE model with KL loss for encoding videos into latents and decoding latent representations into videos. Used in [Allegro](https://github.com/rhymes-ai/Allegro). This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Parameters: in_channels (int, defaults to `3`): Number of channels in the input image. out_channels (int, defaults to `3`): Number of channels in the output. down_block_types (`Tuple[str, ...]`, defaults to `("AllegroDownBlock3D", "AllegroDownBlock3D", "AllegroDownBlock3D", "AllegroDownBlock3D")`): Tuple of strings denoting which types of down blocks to use. up_block_types (`Tuple[str, ...]`, defaults to `("AllegroUpBlock3D", "AllegroUpBlock3D", "AllegroUpBlock3D", "AllegroUpBlock3D")`): Tuple of strings denoting which types of up blocks to use. block_out_channels (`Tuple[int, ...]`, defaults to `(128, 256, 512, 512)`): Tuple of integers denoting number of output channels in each block. temporal_downsample_blocks (`Tuple[bool, ...]`, defaults to `(True, True, False, False)`): Tuple of booleans denoting which blocks to enable temporal downsampling in. latent_channels (`int`, defaults to `4`): Number of channels in latents. layers_per_block (`int`, defaults to `2`): Number of resnet or attention or temporal convolution layers per down/up block. act_fn (`str`, defaults to `"silu"`): The activation function to use. norm_num_groups (`int`, defaults to `32`): Number of groups to use in normalization layers. temporal_compression_ratio (`int`, defaults to `4`): Ratio by which temporal dimension of samples are compressed. sample_size (`int`, defaults to `320`): Default latent size. scaling_factor (`float`, defaults to `0.13235`): The component-wise standard deviation of the trained latent space computed using the first batch of the training set. This is used to scale the latent space to have unit variance when training the diffusion model. The latents are scaled with the formula `z = z * scaling_factor` before being passed to the diffusion model. When decoding, the latents are scaled back to the original scale with the formula: `z = 1 / scaling_factor * z`. For more details, refer to sections 4.3.2 and D.1 of the [High-Resolution Image Synthesis with Latent Diffusion Models](https://huggingface.co/papers/2112.10752) paper. force_upcast (`bool`, default to `True`): If enabled it will force the VAE to run in float32 for high image resolution pipelines, such as SD-XL. VAE can be fine-tuned / trained to a lower range without losing too much precision in which case `force_upcast` can be set to `False` - see: https://huggingface.co/madebyollin/sdxl-vae-fp16-fix TrXrYr.rrrrlatent_channelsrrrtemporal_compression_ratio sample_sizescaling_factor force_upcastrNc t|t|||||| | | d |_t |||||| | | |_t jd|zd|zd|_t j||d|_ d|_ d|_ dt|dz z|_ d|_d|_d |_d }|| | f|_||jz | |jz | |j z f|_y) NT) rXrYrrrrrrr)rXrYrrrrrrr rFxP)r#r$rencoderrdecoderr&r quant_convpost_quant_conv use_slicing use_tilingrspatial_compression_ratiotile_overlap_ttile_overlap_htile_overlap_wkernelr)r0rXrYrrrrrrrrrrrrr sample_framesr4s r5r$zAutoencoderKLAllegro.__init__s: '#(-'A1-+   ('%)%=1-+   ))A$7_9LaP!yy/1M !)*s3E/F/J)K&!  $k;? D// / $-- - $-- -  r6xcR|jr|j|Std)Nz5Encoding without tiling has not been implemented yet.)r tiled_encodeNotImplementedError)r0rs r5_encodezAutoencoderKLAllegro._encode( ??$$Q' '!"YZZr6 return_dictc(|jrU|jddkDrC|jdDcgc]}|j|}}t j |}n|j|}t |}|s|fSt|Scc}w)a Encode a batch of videos into latents. Args: x (`torch.Tensor`): Input batch of videos. return_dict (`bool`, defaults to `True`): Whether to return a [`~models.autoencoder_kl.AutoencoderKLOutput`] instead of a plain tuple. Returns: The latent representations of the encoded videos. If `return_dict` is True, a [`~models.autoencoder_kl.AutoencoderKLOutput`] is returned, otherwise a plain `tuple` is returned. rr) latent_dist)rrFsplitr r;r<rr)r0rr x_sliceencoded_slicesh posteriors r5encodezAutoencoderKLAllegro.encode's"    QCD771:Ndll73NNN .)A QA03 < "y99OsBzcR|jr|j|Std)Nz5Decoding without tiling has not been implemented yet.)r tiled_decoder)r0rs r5_decodezAutoencoderKLAllegro._decodeDr r6c|jrU|jddkDrC|jdDcgc]}|j|}}t j |}n|j|}|s|fSt |Scc}w)a Decode a batch of videos. Args: z (`torch.Tensor`): Input batch of latent vectors. return_dict (`bool`, defaults to `True`): Whether to return a [`~models.vae.DecoderOutput`] instead of a plain tuple. Returns: [`~models.vae.DecoderOutput`] or `tuple`: If return_dict is True, a [`~models.vae.DecoderOutput`] is returned, otherwise a plain `tuple` is returned. rrr)rrFrrr;r<r )r0rr z_slicedecoded_slicesdecodeds r5decodezAutoencoderKLAllegro.decodeLsv    QCD771:Ndll73NNNii/Gll1oG: G,,OsBc  d}|j}|jj}|j\}}}}} t j ||j dz |jdz dz} t j ||j dz |jdz dz} t j | |j dz |jdz dz} d} |j| | z| zd|jjz|j d|z|j d|z|j d|zf}|j|||j d|j d|j df}t| D]}t| D]u}t| D]c}||jdz||jdz|j dz}}||jdz||jdz|j dz}}||jdz||jdz|j dz}}|dddd||||||f}||| |z<| |z|dz k(s| | | z| zdz k(r|j|}| | | z| zdz k(r | |z|dz k7r|d| |zdz|| | |zz dn||| |z dz| dz|j|||j d|j d|j df}| dz } fx|j|d|jjz||z||z| |zf}|j d|z|j d|z|j d|zf}|jd|z|jd|z|jd|zf}|d|dz |d|dz |d|dz f}t| D]}||dz||dz|dz}}t| D]}||dz||dz|dz}}t| D]u}||dz||dz|dz}}t|| |df|| |df|| |df||| z| z|| zz|zjd}|dddd||||||fxx|z cc<w|jdddddjdd}|j!|}|j#d|dfjddddd}|SNrrr rr@r:)rconfigrrFmathfloorrr new_zerosrrqr_prepare_for_blend unsqueezerDrGrrC)r0rlocal_batch_sizersrtr>r num_framesheightwidthoutput_num_frames output_height output_widthcount output_latentvae_batch_inputryjkn_startn_endh_starth_endw_startw_end video_cubelatent output_kernel output_strideoutput_overlap latent_means r5rz!AutoencoderKLAllegro.tiled_encodefs  + + [[ 3 3>?gg; L*fe JJ T[[^(Ct{{ST~'UVYZZ FT[[^$;t{{1~#MNQRR zz54;;q>#9T[[^"KLqP  !M1L@DKK/// A"$ A"$ A"$   ++'7t{{ST~W[WbWbcdWegkgrgrstgu&vw() A=) |,A%&Q%7T[[^9KdkkZ[n9\UG%&Q%7T[[^9KdkkZ[n9\UG%&Q%7T[[^9KdkkZ[n9\UG!"1a wu}#T!UJ@JOE,<$<= 004Dq4HH $5 $E $TWX$XX!%o!>"%6%F%UXY%YY %(8 8!+<=L.*;<%a-&7,&F\IY&Y\]&]^hhijk #K 1a wu}LMQ\\M] ] ]1aA.66q!<(!!!j"%56>>q!Q1M r6c  d}|j}|jj}|jd|z|jd|z|jd|zf}|jd|z|jd|z|jd|zf}|j \}}} } } |j dddddjdd}|j|}|jd|dfj ddddd}tj| |dz |dz dz} tj| |dz |dz dz} tj| |dz |dz dz}d}|j| | z|z|jj|jd|jd|jdf}|j|||d|d|df}t| D] }t| D]}t|D]}||dz||dz|dz}}||dz||dz|dz}}||dz||dz|dz}}|dddd||||||f}||||z<||z|dz k(s|| | z|zdz k(rl|j|}|| | z|zdz k(r ||z|dz k7r|d||zdz||||zz dn||||z dz|dz|j|||d|d|df}|dz }|j||jj| |z| |z| |zf}|jd|jdz |jd|jdz |jd|jdz f}t| D]}||jdz||jdz|jdz}}t| D]}||jdz||jdz|jdz}}t|D]}||jdz||jdz|jdz}}t!|| |df|| |df|||df||| z|z||zz|zj#d}|dddd||||||fxx|z cc<|j dddddj%}|Sr)rr rrrrFrDrGrrCr!r"r#rYrqrr$r% contiguous) r0rr&r'r( latent_kernel latent_strider>rr)r*r+r,r-r.r/decoded_videosr1ryr2r3r4r5r6r7r8r9current_latent current_videovideo video_overlapout_video_blends r5rz!AutoencoderKLAllegro.tiled_decodes  + + [[ 3 3 A",dkk!n.BDKKPQNVXDXX  A",dkk!n.BDKKPQNVXDXX >?gg; L*fe IIaAq! $ , ,Q 2   # KKJ+ , 4 4Q1a C JJ ]15E(EWXIY'YZ]^^ F]1-=$=qAQ#QRUVV zz5=+;#;}Q?O"OPSTT !M1L@ (( A A A   ++ |]1-=}Q?OQ^_`Qa b () A=) |,A%&q)9%91}Q?O;OR_`aRb;bUG%&q)9%91}Q?O;OR_`aRb;bUG%&q)9%91}Q?O;OR_`aRb;bUG%&q!WU]GEM7SX='X%YN@NOE,<$<= 004Dq4HH $5 $E $TWX$XX(, _(E "%6%F%UXY%YY %(8 8%*:":Q">RN55;K3K+K+MNXeN53C+Ca+G%RS)T*+++-|]1=M}]^O_anopaqr+QJE9  @ Z)A)A:PR?TZ]_T_afikaklm KKNT[[^ + KKNT[[^ + KKNT[[^ + () `AQ/T[[^1CdkkRSn1TUG=) `!"T[[^!3QQ5G$++VW.5X|,`A%&Q%7T[[^9KdkkZ[n9\UG&8-}Q/?@M=+;<L-*:;&q='8<'G!lJZ'Z]^'^_iijkl 'O !Q wu}gemKLP__L` ` ` aAq!,779 r6rsample_posterior generatorc|}|j|j}|r|j|}n|j}|j |j}|s|fSt |S)a Args: sample (`torch.Tensor`): Input sample. sample_posterior (`bool`, *optional*, defaults to `False`): Whether to sample from the posterior. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`DecoderOutput`] instead of a plain tuple. generator (`torch.Generator`, *optional*): PyTorch random number generator. )rKr)rr rmoderr ) r0rrJr rKrrrdecs r5rIzAutoencoderKLAllegro.forwardsf" KKN..   9 5A Akk!n##6MC((r6)rrrrrrrr@r rrKr@i@gp= ף?T)T)FTN)rLrMrNrO _supports_gradient_checkpointingrr%rrrQrPr$r;rSr r rrrrrr rrrr GeneratorrIrTrUs@r5rrs+Z(,$- + /C7Q5O !!,- $!5G G G  S/ G c3hG "#s(O!G "%*$)$4#G $#(c "2%G &'G ()G *+G ,-G .%*/G 01G 23G 45G 6 7G G R[[%,,[37::,0: "E*F$GG H::8[[%,,[- -4-5X]XdXdIdCe--2QellQu||QfWellWu||Wx"' /3 ) )) ) EOO, ) }ell* + )r6rc|\}}}|\}}} |\} } } |dkDr|dkDrx|ddddd|ddddftjd|jj|j|z j |ddz|ddddd|ddddf<||dz kr}|dddd| dddddfdtjd|jj|j|z z j |ddz|dddd| dddddf<|dkDrw|ddddddd| ddftjd| jj|j| z j | dz|ddddddd| ddf<||dz kr||dddddd| dddfdtjd| jj|j| z z j | dz|dddddd| dddf<| dkDrg|ddddddddd| ftjd| jj|j| z z|ddddddddd| f<| | dz krl|dddddddd| dfdtjd| jj|j| z z z|dddddddd| df<|S)Nrr)r;arangerPrdevicereshape) n_paramh_paramw_paramrnn_max overlap_nrh_max overlap_hww_max overlap_ws r5r$r$1s!Aui!Aui!Aui1} q5)*1a9a+B)C Q *00255ahh?)KgiA&*'AaAiKA% & uqy=)*1a)a+B)CELLI.44699!((CiOOgiA&*'AaYJKA% & 1u%&q!Q) Q'>%? LLI & , , . 1 1!(( ;i G ')Q & !Q1Y; !" 519}%&q!Q Q'>%?  Q *00255ahh?)K K ')Q & !QI:; !" 1u%&q!Q1Y;'>%? LLI & , , . 1 1!(( ;i G& !Q1a k !" 519}%&q!QI:;'>%?  Q *00255ahh?)K K& !Q1yjk !" Hr6)(r!typingrrrr;torch.nnr&configuration_utilsrrutils.accelerate_utilsr attention_processorr r autoencoders.vaer r downsamplingrmodeling_outputsrmodeling_utilsrr~r upsamplingrvaerModulerrWrrrrrr$r6r5rms  )) B88J'2'"#!bryybJTTnOryyOdlBIIl^|ryy|~DryyDNI):'7I)X  r6