i2 VddlmZddlmZmZmZmZmZmZddl Z ddl m Z ddl m Z mZddlmZmZddlmZmZmZmZmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&ejNe(Z)eGddeZ*Gdde!ee eeeZ+Gdde!e eeeZ,y)) dataclass)AnyDictListOptionalTupleUnionN) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin)USE_PEFT_BACKEND BaseOutputloggingscale_lora_layersunscale_lora_layers)AttentionMixin) CacheMixin) zero_module)Transformer2DModelOutput) ModelMixin) QwenEmbedRopeQwenImageTransformerBlockQwenTimestepProjEmbeddingsRMSNormc2eZdZUeej ed<y)QwenImageControlNetOutputcontrolnet_block_samplesN)__name__ __module__ __qualname__rtorchTensor__annotations__{/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/diffusers/models/controlnets/controlnet_qwenimage.pyrr(s#ELL11r(rceZdZdZe ddededeededededed eeeefd effd Ze ddededed efd Z dd e jde jde de jde jde jdeeeeeefdeeedeeeefdedee j*effdZxZS)QwenImageControlNetModelT patch_size in_channels out_channels num_layersattention_head_dimnum_attention_headsjoint_attention_dimaxes_dims_ropeextra_condition_channelsc t ||xs||_||z|_t dt |d|_t|j|_t|d|_ tj||j|_ tj||j|_tjt!|D cgc]} t#|j||c} |_tjg|_t!t)|j$D]N} |j&j+t-tj|j|jPt-t.jj|| z|j|_d|_ycc} w) Ni'T)thetaaxes_dim scale_rope) embedding_dimgư>)eps)dimr1r0F)super__init__r. inner_dimrlist pos_embedrtime_text_embedrtxt_normnnLinearimg_intxt_in ModuleListrangertransformer_blockscontrolnet_blockslenappendrr$controlnet_x_embeddergradient_checkpointing) selfr,r-r.r/r0r1r2r3r4_ __class__s r)r=z!QwenImageControlNetModel.__init__2sj (7K,/AA&UT.=Q^bc9W 3> ii T^^< ii 3T^^D "$--z*    *(;'9  # "$r!2s42234 bA  " " ) )+biiPTP^P^6_*` a b%0 HHOOK*BBDNN S& "',#% sG ct|j}||d<||d<||d<||d<|j|}|r|jj |jj |j j |j j |jj |jj |jj |jj |jj |jj dt|j|_ |S)Nr/r0r1r4F)strict) dictconfig from_configr@load_state_dict state_dictrArErFrIrrM) cls transformerr/r0r1load_weights_from_transformerr4rU controlnets r)from_transformerz)QwenImageControlNetModel.from_transformerask(())|'9#$(;$%-E)*__V, (  0 01F1F1Q1Q1S T  & & 6 6{7R7R7]7]7_ `    - -k.@.@.K.K.M N    - -k.@.@.K.K.M N  ) ) 9 9+:X:X:c:c:ens 9 t/::;[;[/\J ,r( hidden_statescontrolnet_condconditioning_scaleencoder_hidden_statesencoder_hidden_states_masktimestep img_shapes txt_seq_lensjoint_attention_kwargs return_dictreturnc | #| j} | jdd} nd} tr t|| n)| '| j ddt j d|j|}||j|z}|j||} |j|||j} |j|j}|j|}|j|}d}t!|j"D]V\}}t%j&r&|j(r|j+||||| | \}}n||||| | | \}}||fz}Xd}t-||j.D]\}}||}||fz}|Dcgc]}||z }}t1|dk(rdn|}tr t3|| | s|St5| Scc}w) am The [`FluxTransformer2DModel`] forward method. Args: hidden_states (`torch.FloatTensor` of shape `(batch size, channel, height, width)`): Input `hidden_states`. controlnet_cond (`torch.Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): The scale factor for ControlNet outputs. encoder_hidden_states (`torch.FloatTensor` of shape `(batch size, sequence_len, embed_dims)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. pooled_projections (`torch.FloatTensor` of shape `(batch_size, projection_dim)`): Embeddings projected from the embeddings of input conditions. timestep ( `torch.LongTensor`): Used to indicate denoising step. block_controlnet_hidden_states: (`list` of `torch.Tensor`): A list of tensors that if specified are added to the residuals of transformer blocks. joint_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. Nscale?z\Passing `scale` via `joint_attention_kwargs` when not using the PEFT backend is ineffective.)devicer')r^rarbtembimage_rotary_embrfr)r )copypoprrgetloggerwarningrErMrAr@rltodtyperBrF enumeraterIr$is_grad_enabledrN_gradient_checkpointing_funcziprJrKrr)rOr^r_r`rarbrcrdrerfrg lora_scalermrn block_samples index_blockblockr block_samplecontrolnet_blocksamples r)forwardz QwenImageControlNetModel.forward}s&V " -%;%@%@%B "/33GSAJJ  dJ /%16L6P6PQXZ^6_6kr M2 &(B(B?(SS ##Hm<>>*l=K_K_>`;;}223 $ .C D $ ,A B "+D,C,C"D = K$$&4+F+F7;7X7X!).$ 84%}8="/*?/I%5+A 84%}*],<%(.237%);?,0;? p ||p p " p %|| p %*LL p ""p T%S# "678p tCy)p !)c3h 8p p  u  ":: ;p r(r+ceZdZdZfdZ ddej deejdee dejdejdejd e ee eeefd e eed e eeefd ed eee ffdZxZS)QwenImageMultiControlNetModela `QwenImageMultiControlNetModel` wrapper class for Multi-QwenImageControlNetModel This module is a wrapper for multiple instances of the `QwenImageControlNetModel`. The `forward()` API is designed to be compatible with `QwenImageControlNetModel`. Args: controlnets (`List[QwenImageControlNetModel]`): Provides additional conditioning to the unet during the denoising process. You must set multiple `QwenImageControlNetModel` as a list. cVt|tj||_y)N)r<r=rCrGnets)rO controlnetsrQs r)r=z&QwenImageMultiControlNetModel.__init__s MM+. r(r^r_r`rarbrcrdrerfrgrhc 8t|jdk(rr|jd} tt||D]I\} \} }| || ||||||| |  }| dk(r|}#|&)t||Dcgc] \}}||z }}}KSt dcc}}w)Nr) r^r_r`rarbrcrdrerfrgzJQwenImageMultiControlNetModel only supports a single controlnet-union now.)rKrrvry ValueError)rOr^r_r`rarbrcrdrerfrgr\iimagerjr{control_block_samplescontrol_block_sampler~s r)rz%QwenImageMultiControlNetModel.forwards tyy>Q 1J%.s?DV/W%X !>E5 *"/$)',*?/I%)!-+A + ! 6,9)$05J5VGJJ_anFo1 B 4l1<?1-1' 4%$ij j 1s5B)NNNNNNT)r!r"r#__doc__r=r$rrtensorrr%rrrrrrrrr rrrrs@r)rrs //337%);?,0;? ,%((,%ell+,%!K ,% %|| ,% %*LL ,%"",%T%S# "678,%tCy),%!)c3h 8,%,% (%/ 0,%r(r)- dataclassesrtypingrrrrrr r$torch.nnrCconfiguration_utilsr r loadersr rutilsrrrrr attentionr cache_utilsrcontrolnets.controlnetrmodeling_outputsrmodeling_utilsr"transformers.transformer_qwenimagerrrr get_loggerr!rrrr+rr'r(r)rs":: B?bb&$07'   H % 2 2 2@  -=?UWa@ F=%J =MOegq=%r(