i">ddlZddlZddlmZddlmZmZmZmZddl m Z m Z m Z m Z ddlmZerddlmZeGddZy) N) dataclass) TYPE_CHECKINGLiteralOptionalUnion)DeepSpeedSequenceParallelConfigDistributedTypeTorchContextParallelConfigTorchTensorParallelConfig)is_torch_version) Acceleratorc>eZdZUdZdZeeed<dZeeed<dZ eeed<dZ eeed<dZ e ded<dZ eeed <dZe d ed <dZedefed <dZedefed <dZedefed<dZdZdZedZedZedZedZedZedZedZ edZ!edZ"edZ#edZ$edZ%edZ&edZ'de(fd Z)d*dee(fd!Z*d"e+e+ed#fe+e(d#fffd$Z,d%Z-d&e(d'efd(Z.d+d)Z/y),ParallelismConfiga A dataclass to configure parallelisms applied to the model. Inspired by torchtitan's `ParallelDims` https://github.com/pytorch/torchtitan/blob/main/torchtitan/distributed/parallel_dims.py Args: dp_replicate_size (`int`, defaults to `1`): The size of the data parallel group. If `dp_replicate_size` is set to 1, the data parallel replication group will not be used. dp_shard_size (`int`, defaults to `1`): The size of the model shard group. If `dp_replicate_size > 1` and `tp_size > 1`, `dp_shard_size` must also be greater than 1, as composing DDP + TP is currently not supported. tp_size (`int`, defaults to `1`): The size of the tensor parallel group. If `tp_size` is set to `1`, the tensor parallel group will not be used. tp_handler (`~utils.TorchTensorParallelConfig`, defaults to `None`): The handler for the tensor parallel group. cp_size (`int`, defaults to `1`): The size of the context parallel group. Currently not supported, but reserved for future use and enabled for downstream libraries. cp_backend (`str`, defaults to `torch`): Which CP backend to use: `torch` (FSDP2) sp_size (`int`, defaults to `1`): The size of the sequence parallel group. sp_backend (`str`, defaults to `deepspeed`): Which SP backend to use:`deepspeed` (ALST/Ulysses) You may obtain different distributed data parallel paradigms by configuring `dp_replicate_size` and `dp_shard_size` together: - `dp_replicate_size == 1` and `dp_shard_size > 1`, we obtain Fully Sharded Data Parallel (FSDP). - `dp_replicate_size > 1` and `dp_shard_size > 1`, we obtain Hybrid Sharded Data Parallel (HSDP). - `dp_replicate_size > 1` and `dp_shard_size == 1` is an invalid configuration, to use pure DP, use `DistributedDataParallelKwargs` instead. Ndp_replicate_size dp_shard_sizetp_sizecp_sizetorch cp_backendsp_size deepspeed sp_backend tp_handler cp_handler sp_handlerc d|jd|jd|jd|jd|jd|j d|j d|jd |jd |jd S) Nz'ParallelismConfig( dp_replicate_size=z, dp_shard_size=z , tp_size=z , cp_size=z, cp_backend=z , sp_size=z, sp_backend=z, total_size=z tp_handler=z, cp_handler=z) ) rrrrrrr total_sizerrselfs g/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/accelerate/parallelism_config.py__repr__zParallelismConfig.__repr__Us ##'#9#9":;#1123'' OO,-' OO,- OO,- OO,- OO,C 1 c ddl}dg}|j|jjDcic]3\}}||vr*|t |dr|j|jn|5c}}ycc}}w)Nr device_mesh__dict__)copydeepcopyr%itemshasattr)rr&_non_serializable_fieldskvs r to_jsonzParallelismConfig.to_jsondsm$1?  !MM//1 Aq44:0F4==,AM   s8A1 cRg}|jr|dgz }|jr|dgz }|S)zENames of enabled dimensions across which data parallelism is applied. dp_replicatedp_shard)dp_replicate_enableddp_shard_enabledrdimss r dp_dim_nameszParallelismConfig.dp_dim_namesqs9  $ $ ^$ $D  ZL D r"cvg}|jr|dgz }|jr|dgz }|jr|dgz }|S)z]Names of enabled dimensions which will receive the same batch (non-data parallel dimensions).tpcpsp) tp_enabled cp_enabled sp_enabledr3s r non_dp_dim_namesz"ParallelismConfig.non_dp_dim_names{sD ?? TFND ?? TFND ?? TFND r"cRg}|jr|dgz }|jr|dgz }|S)zlNames of enabled dimensions which will be flattened into a joint mesh across which is model sharded in FSDP.r0r8)r2r;r3s r dp_shard_cp_dim_namesz'ParallelismConfig.dp_shard_cp_dim_namess5  ZL D ?? TFND r"cvg}|jr|dgz }|jr|dgz }|jr|dgz }|S)z@Names of enabled dimensions across which loss should be averagedr/r0r8)r1r2r;r3s r dp_cp_dim_namesz!ParallelismConfig.dp_cp_dim_namessK  $ $ ^$ $D  ZL D ?? TFND r"c:g}|jr|dgz }|dgz }|S)z^Names of enabled dimensions across which FSDP is applied, including data parallel replication.r/ dp_shard_cp)r1r3s r fsdp_dim_namesz ParallelismConfig.fsdp_dim_namess0  $ $ ^$ $D  r"c|j|jz|jz|jz|jzS)zSThe total size of the parallelism configuration, which is the product of all sizes.)rrrrrrs r rzParallelismConfig.total_sizes9%%(:(::T\\IDLLX[_[g[gggr"cN|j|jz|jzS)zhThe size of the non-data parallel dimensions, which is the product of tensor and context parallel sizes.)rrrrs r non_data_parallel_sizez(ParallelismConfig.non_data_parallel_sizes ||dll*T\\99r"c4|j|jzS)z_The size of the data parallel dimensions, which is the product of data parallel replication and)rrrs r data_parallel_sizez$ParallelismConfig.data_parallel_sizes%%(:(:::r"c |jdkDS)zKTrue if data parallel replication is enabled, i.e. `dp_replicate_size > 1`.)rrs r r1z&ParallelismConfig.dp_replicate_enableds%%))r"c |jdkDS)zDTrue if data parallel sharding is enabled, i.e. `dp_shard_size > 1`.rK)rrs r r2z"ParallelismConfig.dp_shard_enableds!!A%%r"c |jdkDS)z:True if tensor parallelism is enabled, i.e. `tp_size > 1`.rK)rrs r r:zParallelismConfig.tp_enabled||ar"c |jdkDS)z;True if context parallelism is enabled, i.e. `cp_size > 1`.rK)rrs r r;zParallelismConfig.cp_enabledrNr"c |jdkDS)z;True if context parallelism is enabled, i.e. `sp_size > 1`.rK)rrs r r<zParallelismConfig.sp_enabledrNr"c4|j|jzS)z$Names of all active mesh dimensions.)r5r=rs r active_mesh_dimsz"ParallelismConfig.active_mesh_dimss  4#8#888r" device_typectddrddlm}n td|j }t |dk(ry|\}}||||}|j r||j jd|jr||jjd |jr||jjd |S) a!Builds a device mesh for the given device type based on the parallelism configuration. This method will also create required joint meshes (e.g. `dp_shard_cp`, `dp_cp`, `dp`). Args: device_type (`str`): The type of device for which to build the mesh, e z>=z2.2.0r)init_device_meshz4Building a device_mesh requires to have torch>=2.2.0N)mesh_dim_namesdprCdp_cp) r torch.distributed.device_meshrU RuntimeError _get_meshlenr5_flattenr?rA)rrSrUmeshrV mesh_shaper$s r build_device_meshz#ParallelismConfig.build_device_meshs D' * FUV V~~ t9>%)" &  )    )) * 3 3D 9  % % 22 3 < <] K    ,, - 6 6w ?r"c|j&|"|j||_|jSd|?|jj|k7r&td|jjd|d|jS)Nz@You need to pass a device_type e.g cuda to build the device meshz4The device_mesh is already created with device type z@. However, you are trying to get a device mesh with device_type z<. Please check if you correctly initialized your device_mesh)r$r`rS ValueError)rrSs r get_device_meshz!ParallelismConfig.get_device_meshs    #&#'#9#9+#F ZZ&##//;>$NtO_O_OkOkNlmmnymzzvwr"return.c|jDcic]}||j|}}gdt|jfd}t t |Scc}w)zQGenerate mesh shape and dimension names for torch.distributed.init_device_mesh().)r/r0r8r9r7c,j|dS)Nr)index)x mesh_orders r z-ParallelismConfig._get_mesh.. s:++AaD1r")key)rR_sizessortedr(tuplezip)r parallelism mesh_dims sorted_itemsris @r r[zParallelismConfig._get_meshseOSNcNcd{[$++k"::d dD  OO 2 S,'((esAc .|j.ttjj dd|_|j .ttjj dd|_|j .ttjj dd|_|j.ttjj dd|_|j%tjj dd|_|j.ttjj dd|_ |j%tjj d d |_ |j d kDr|jt|_ |jd kDr|jt|_ nqtt }t!|j||js>t#d |jd||jdt%|j|jd kDr|j&t)|_|jd krt#d|j|j d krt#d|j |j d krt#d|j |jd krt#d|jdg}|j|vrt#d|d|j|jd krt#d|jd g}|j|vrt#d|d|j|j d kDs|jd kDr)|jd kDr|j d k(r t#d|j|j |j |j|jd|_y)N$PARALLELISM_CONFIG_DP_REPLICATE_SIZE1 PARALLELISM_CONFIG_DP_SHARD_SIZEPARALLELISM_CONFIG_TP_SIZEPARALLELISM_CONFIG_CP_SIZEPARALLELISM_CONFIG_CP_BACKENDrPARALLELISM_CONFIG_SP_SIZEPARALLELISM_CONFIG_SP_BACKENDrrK)rzParallelismConfig's cp_backend=z requires z, but cp_handler was set to z.dp_replicate_size must be at least 1, but got z*dp_shard_size must be at least 1, but got z$tp_size must be at least 1, but got z$cp_size must be at least 1, but got zcp_backend must be one of z , but got z$sp_size must be at least 1, but got zsp_backend must be one of aCTensor/Context parallelism (tp/cp_size > 1) cannot be used with pure data parallelism (dp_replicate_size > 1 and dp_shard_size == 1). Please set dp_shard_size > 1 and dp_replicate_size == 1 to compose FSDP + TP/CP for 2D parallel, or set dp_replicate_size == 1 and dp_shard_size > 1 to compose HSDP + TP/CP for 3D parallel.)r/r0r7r8r9)rintosenvirongetrrrrrrrr rr dict isinstancerbtyperrrl)rcp_backends_config_mapvalid_cp_backendsvalid_sp_backendss r __post_init__zParallelismConfig.__post_init__ s  ! ! )%(8^`c)d%eD "    %!$RZZ^^4VX[%\!]D  << rzz~~.JCPQDL << rzz~~.JCPQDL ?? " jjnn-LgVDO << rzz~~.JCPQDL ?? " jjnn-LkZDO <)-4*&"$//3I$//3Z[$9$//9J*Uklpl{l{U|T}~Z[_`d`o`o[pZqr <RS S <RS S$I ??"3 39:K9LJW[WfWfVghi i <RS S(M ??"3 39:K9LJW[WfWfVghi i LL1  q 0d6L6Lq6PUYUgUgklUlo  !22**,,,,,,   r"rpsizec||jjvs"Jd|jj||j|<t||d|y)NzParallelism must be one of _size)rlkeyssetattr)rrprs r _set_sizezParallelismConfig._set_sizeQs\dkk..00d4OPTP[P[P`P`PbOc2dd0#' K  U+T2r"ct}|js|jdk(ry|jdk(r|jd|j|j|jk7r&t d|jd|jd|jdkDrN|j sB|js6|jtjk(st d|jd|jjD]4\}}|dk(s t||dd|jd |d |d 6|r:|jr-tj d d j#|zt$yyy)NrKr/zParallelismConfig total_size (z ) does not match num_processes (zJ). Please adjust dp_replicate_size/ dp_shard_size/tp_size/cp_size/sp_size.zParallelismConfig is only compatible DistributedType.FSDP (version 2) or DistributedType.Multi{Device} or DistributedType.DEEPSPEED, but got ._handlerzParallelismConfig.z_handler is set, but z0_size is set to 1. This handler will be ignored.z.ParallelismConfig has the following warnings:  )set multi_devicerr num_processesrbis_fsdp2distributed_typer DEEPSPEEDrlr(getattraddis_main_processwarningswarnjoin UserWarning)r accelerator _warningsrprs r _validate_acceleratorz'ParallelismConfig._validate_acceleratorVsE ''DOOq,@  ??a  NN>;+D+D E ??k77 700AB""-";";!<=9:  ??Q   ''++/H/HHbcnccb@@AB "&!2!2!4  KqyWTk](+CTJV ( 5J;-XHI  44 MMADIIiDXX 59r")N)rr )0__name__ __module__ __qualname____doc__rrr|__annotations__rrrrrrrrrr rr rrr$r!r-propertyr5r=r?rArDrrGrIr1r2r:r;r<rRstrr`rcrnr[rrrr"r rr!s:!F(,x}+#'M8C='!GXc]!!GXc]!#'J '!GXc]!'+J $+:>Jd556=:>Jd667>?CJd;;<CK        hh::;;**&&      99S<  8C=   )5sCx%S/!AB )B H3S33 $r"r)r}r dataclassesrtypingrrrraccelerate.utils.dataclassesrr r r accelerate.utils.versionsr accelerater rrr"r rsI !:: 7& XX Xr"