iQ ddlZddlmZmZmZddlZddlmZddlm Z m Z ddl m Z ddl mZddlmZddlmZd d lmZd d lmZmZmZmZmZmZd d lmZd d lmZddl m!Z!m"Z"m#Z#m$Z$m%Z%Gddee!ee ee Z&y)N)OptionalTupleUnion) ConfigMixinregister_to_config)PeftAdapterMixin)FromOriginalModelMixin) deprecate)apply_forward_hook)AttentionMixin)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORS AttentionAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0)AutoencoderKLOutput) ModelMixin)AutoencoderMixinDecoder DecoderOutputDiagonalGaussianDistributionEncoderc&eZdZdZdZddgZe d/dededee dfd ee dfd eedfd ed e d ededede de de de de de deeeff"fd Z dZ dejdejfdZe d0dejde deeeeffdZd0dejde deeejffdZe d1dej0de deeej0ffd Zd!ejd"ejd#edejfd$Zd!ejd"ejd#edejfd%Zdejdejfd&Zd0dejde defd'Zd0dejde deeejffd(Z d2d)ejd*e de d+eej@deeejff d,Z!d-Z"d.Z#xZ$S)3AutoencoderKLFlux2a A VAE model with KL loss for encoding images into latents and decoding latent representations into images. This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Parameters: in_channels (int, *optional*, defaults to 3): Number of channels in the input image. out_channels (int, *optional*, defaults to 3): Number of channels in the output. down_block_types (`Tuple[str]`, *optional*, defaults to `("DownEncoderBlock2D",)`): Tuple of downsample block types. up_block_types (`Tuple[str]`, *optional*, defaults to `("UpDecoderBlock2D",)`): Tuple of upsample block types. block_out_channels (`Tuple[int]`, *optional*, defaults to `(64,)`): Tuple of block output channels. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. latent_channels (`int`, *optional*, defaults to 4): Number of channels in the latent space. sample_size (`int`, *optional*, defaults to `32`): Sample input size. force_upcast (`bool`, *optional*, default to `True`): If enabled it will force the VAE to run in float32 for high image resolution pipelines, such as SD-XL. VAE can be fine-tuned / trained to a lower range without losing too much precision in which case `force_upcast` can be set to `False` - see: https://huggingface.co/madebyollin/sdxl-vae-fp16-fix mid_block_add_attention (`bool`, *optional*, default to `True`): If enabled, the mid_block of the Encoder and Decoder will have attention blocks. If set to false, the mid_block will only have resnet blocks TBasicTransformerBlock ResnetBlock2D in_channels out_channelsdown_block_types.up_block_typesblock_out_channelslayers_per_blockact_fnlatent_channelsnorm_num_groups sample_size force_upcastuse_quant_convuse_post_quant_convmid_block_add_attentionbatch_norm_epsbatch_norm_momentum patch_sizec t|t||||||| d| |_t |||||| |||_| rt jd|zd|zdnd|_| rt j||dnd|_ t jtj||z||dd|_ d|_d|_|j j"|_t'|j j"t(t*fr|j j"dn|j j"} t-| dt/|j j0dz zz |_d |_y) NT) r!r"r#r%r&r'r)double_zr.)r!r"r$r%r&r)r'r.r rF)epsmomentumaffinetrack_running_statsrg?)super__init__rencoderrdecodernnConv2d quant_convpost_quant_conv BatchNorm2dmathprodbn use_slicing use_tilingconfigr*tile_sample_min_size isinstancelisttupleintlenr%tile_latent_min_sizetile_overlap_factor)selfr!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1 __class__s |/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/diffusers/models/autoencoders/autoencoder_kl_flux2.pyr9zAutoencoderKLFlux2.__init__HsiH #(-1-+$;   '%)1-+$;   Uc"))A$7_9LaPhlQdryy/1Mjn.. IIj !O 3( $  !%)KK$;$;!$++11D%=A KK # #A &((  %( qSA_A_=`cd=d7e(f$g!#' c jtd|jjDr t}nmtd|jjDr t }n8t dt t|jj|j|y)ze Disables custom attention processors and sets the default attention implementation. c3@K|]}|jtvywN)rPr.0procs rQ z@AutoencoderKLFlux2.set_default_attn_processor..si4t~~!>>ic3@K|]}|jtvywrU)rPrrVs rQrYz@AutoencoderKLFlux2.set_default_attn_processor..sh$#==hrZzOCannot call `set_default_attn_processor` when attention processors are of type N) allattn_processorsvaluesrr ValueErrornextiterset_attn_processor)rO processors rQset_default_attn_processorz-AutoencoderKLFlux2.set_default_attn_processors i4K_K_KfKfKhi i,.I h$J^J^JeJeJgh h%IabfgklpmAmAmHmHmJhKcLbMN   *rRxreturnc|j\}}}}|jr/||jkDs||jkDr|j|S|j |}|j |j |}|SrU)shaperErG _tiled_encoder:r>)rOre batch_size num_channelsheightwidthencs rQ_encodezAutoencoderKLFlux2._encodesp23''/ L&% ??(A(A AVdNgNgEg%%a( (ll1o ?? &//#&C rR return_dictc(|jrU|jddkDrC|jdDcgc]}|j|}}t j |}n|j|}t |}|s|fSt|Scc}w)a Encode a batch of images into latents. Args: x (`torch.Tensor`): Input batch of images. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a [`~models.autoencoder_kl.AutoencoderKLOutput`] instead of a plain tuple. Returns: The latent representations of the encoded images. If `return_dict` is True, a [`~models.autoencoder_kl.AutoencoderKLOutput`] is returned, otherwise a plain `tuple` is returned. rr latent_dist)rDrhsplitrotorchcatrr)rOrerpx_sliceencoded_slicesh posteriors rQencodezAutoencoderKLFlux2.encodes    QCD771:Ndll73NNN .)A QA03 < "y99OsBzc.|jrK|jd|jkDs|jd|jkDr|j||S|j|j |}|j |}|s|fSt |S)N)rpsample)rErhrM tiled_decoder?r;r)rOr|rpdecs rQ_decodezAutoencoderKLFlux2._decodes ?? d.G.G G177SU;Y]YrYrKr$$QK$@ @    +$$Q'All1o6MC((rRc:|jr_|jddkDrM|jdDcgc]}|j|j}}t j |}n|j|j}|s|fSt|Scc}w)a Decode a batch of images. Args: z (`torch.Tensor`): Input batch of latent vectors. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a [`~models.vae.DecoderOutput`] instead of a plain tuple. Returns: [`~models.vae.DecoderOutput`] or `tuple`: If return_dict is True, a [`~models.vae.DecoderOutput`] is returned, otherwise a plain `tuple` is returned. rrr)rDrhrtrrrurvr)rOr|rp generatorz_slicedecoded_slicesdecodeds rQdecodezAutoencoderKLFlux2.decodes$    QJK''RS*Uwdll73::UNUii/Gll1o,,G: G,,Vs"Bab blend_extentct|jd|jd|}t|D]A}|dddd| |zddfd||z z z|dddd|ddf||z zz|dddd|ddf<C|S)Nr rminrhrange)rOrrrys rQblend_vzAutoencoderKLFlux2.blend_vs1771:qwwqz<@ |$ xAa\MA$5q89Q\AQ=QRUVWXZ[]^`aWaUbfgjvfvUwwAaAqjM xrRct|jd|jd|}t|D]A}|dddddd| |zfd||z z z|dddddd|f||z zz|dddddd|f<C|S)Nrrr)rOrrrres rQblend_hzAutoencoderKLFlux2.blend_h s1771:qwwqz<@ |$ xAaA }q'889Q\AQ=QRUVWXZ[]^`aWaUbfgjvfvUwwAaAqjM xrRc t|jd|jz z}t|j|jz}|j|z }g}t d|j d|D]}g}t d|j d|D]v}|dddd|||jz|||jzf} |j | } |jjr|j| } |j| x|j|g} t|D]\}}g} t|D]d\}} |dkDr|j||dz || |} |dkDr|j||dz | |} | j| ddddd|d|ff| jtj| dtj| d} | S)aEncode a batch of images using a tiled encoder. When this option is enabled, the VAE will split the input tensor into tiles to compute encoding in several steps. This is useful to keep memory use constant regardless of image size. The end result of tiled encoding is different from non-tiled encoding because each tile uses a different encoder. To avoid tiling artifacts, the tiles overlap and are blended together to form a smooth output. You may still see tile-sized changes in the output, but they should be much less noticeable. Args: x (`torch.Tensor`): Input batch of images. Returns: `torch.Tensor`: The latent representation of the encoded videos. rrr rNdim)rKrGrNrMrrhr:rFr,r>append enumeraterrrurv) rOre overlap_sizer row_limitrowsirowjtile result_rows result_rowrns rQriz AutoencoderKLFlux2._tiled_encodes"444D F4q5<<QU AlKDq5<<AE D,GD!!$q!ZiZ)'C"DE F   uyy; < =ii + rRc "d}tdd|dt|jd|jz z}t|j|jz}|j|z }g}t d|j d|D]}g} t d|j d |D]v} |d d d d |||jz| | |jzf} |j| } |jjr|j| } | j| x|j| g} t|D]\}} g} t| D]d\} } |dkDr|j||dz | | |} | dkDr|j| | dz | |} | j| d d d d d |d |ff| jtj | d tj | d }t#|}|s|fSt%| S) a8Encode a batch of images using a tiled encoder. When this option is enabled, the VAE will split the input tensor into tiles to compute encoding in several steps. This is useful to keep memory use constant regardless of image size. The end result of tiled encoding is different from non-tiled encoding because each tile uses a different encoder. To avoid tiling artifacts, the tiles overlap and are blended together to form a smooth output. You may still see tile-sized changes in the output, but they should be much less noticeable. Args: x (`torch.Tensor`): Input batch of images. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.autoencoder_kl.AutoencoderKLOutput`] instead of a plain tuple. Returns: [`~models.autoencoder_kl.AutoencoderKLOutput`] or `tuple`: If return_dict is True, a [`~models.autoencoder_kl.AutoencoderKLOutput`] is returned, otherwise a plain `tuple` is returned. aCThe tiled_encode implementation supporting the `return_dict` parameter is deprecated. In the future, the implementation of this method will be replaced with that of `_tiled_encode` and you will no longer be able to pass `return_dict`. You will also have to create a `DiagonalGaussianDistribution()` from the returned value. tiled_encodez1.0.0F) standard_warnrrr rNrrr)r rKrGrNrMrrhr:rFr,r>rrrrrurvrr)rOrerpdeprecation_messagerrrrrrrrrrmomentsrzs rQrzAutoencoderKLFlux2.tiled_encode?s ( ~  .'+>eT444D F4q5<<QU AlKDq5<<AE D,GD!!$q!ZiZ)'C"DE F   uyy; < =))KQ/09 < "y99rRc t|jd|jz z}t|j|jz}|j|z }g}t d|j d|D]}g}t d|j d|D]v} |dddd|||jz| | |jzf} |j jr|j| } |j| } |j| x|j|g} t|D]\}}g} t|D]d\} } |dkDr|j||dz | | |} | dkDr|j|| dz | |} | j| ddddd|d|ff| jtj| dtj| d}|s|fSt!|S)a Decode a batch of images using a tiled decoder. Args: z (`torch.Tensor`): Input batch of latent vectors. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.vae.DecoderOutput`] instead of a plain tuple. Returns: [`~models.vae.DecoderOutput`] or `tuple`: If return_dict is True, a [`~models.vae.DecoderOutput`] is returned, otherwise a plain `tuple` is returned. rrr rNrr)rKrMrNrGrrhrFr-r?r;rrrrrurvr)rOr|rprrrrrrrrrrrrs rQrzAutoencoderKLFlux2.tiled_decode}s444D F4q5<<QU AlKDq5<<AE D,GD!!$q!ZiZ)'C"DE F   uyy; < =ii +6MC((rRrsample_posteriorrc|}|j|j}|r|j|}n|j}|j |j}|s|fSt |S)aa Args: sample (`torch.Tensor`): Input sample. sample_posterior (`bool`, *optional*, defaults to `False`): Whether to sample from the posterior. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`DecoderOutput`] instead of a plain tuple. )rr)r{rsrmoderr) rOrrrprrerzr|rs rQforwardzAutoencoderKLFlux2.forwardsf KKN..   9 5A Akk!n##6MC((rRcrd|_|jjD]1\}}dt|jj vs(t d|j|_|jD]%}t|ts|jd'|jty)u Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. > [!WARNING] > This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsr]itemsstrrP__name__r_modulesrHrfuse_projectionsrbr)rO_attn_processormodules rQfuse_qkv_projectionsz'AutoencoderKLFlux2.fuse_qkv_projectionss)-%!%!5!5!;!;!= v A~#n66??@@ !tuu v)-(<(<%lln 3F&),''T'2 3  5 78rRcT|j|j|jyy)unDisables the fused QKV projection if enabled. > [!WARNING] > This API is 🧪 experimental. N)rrb)rOs rQunfuse_qkv_projectionsz)AutoencoderKLFlux2.unfuse_qkv_projectionss)  ( ( 4  # #D$A$A B 5rR)rr)DownEncoderBlock2Drrr)UpDecoderBlock2Drrr)rr silu riTTTTg-C6?g?)r r )T)TN)FTN)%r __module__ __qualname____doc__ _supports_gradient_checkpointing_no_split_modulesrrKrrboolfloatr9rdruTensorror rrrr{rr FloatTensorrrrrirrr Generatorrrr __classcell__)rPs@rQrr'sO6(,$0/B- + / !"!!!#$((, $%(&,CT(T(T( S/ T(c3hT( "#s(O!T(,-T(./T(01T(23T(45T(67T(89T(:";T(<"&=T(>?T(@#AT(B#s(OCT(T(n +  %,, 37::,0: "E*F$GG H::8 ) )D )E-Y^YeYeJeDf )HL-""-15- }e/// 0--8%,,cell %,,cell .u||. .`<:ell<:<:I\<:|/)ell/)/)}^c^j^jOjIk/)h"' /3 ) )) ) EOO, ) }ell* + ):9,CrRr)'rAtypingrrrrutorch.nnr<configuration_utilsrrloadersr loaders.single_file_modelr utilsr utils.accelerate_utilsr attentionrattention_processorrrrrrrmodeling_outputsrmodeling_utilsrvaerrrrrrrRrQrs^ )) B'?8&3'``AC .+?UWgACrR