iUdZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z mZdd lmZdd lmZdd lmZmZmZmZmZdd lmZd dlmZddlmZddlm Z m!Z!m"Z"m#Z#m$Z$erddl%m&Z&ndZ&erddl'm(Z(ndZ(ejRe*Z+iZ,e-e.e/efe0d<iZ1e-e.e/efe0d<ee.e.dzfgderdndfderdndfderdndfderdndfderdndfd erdndfd!erd"ndfd#erd$ndfd%d&erdndfd'erd(ndfd)d*d+erd,ndfd-erd.ndfd/d0erd1ndfd2d3erdndfd4erd5ndfd6d7erdndfd8d9erd:ndfd;d<erdndfd=d>erdndfd?erdndfd@dAerdBndfdCerd5ndfdDerdndfdEerdndfdFerdndfdGerdndfdHerdIndfdJdKdLdMdNerd5ndfdOerdPndfdQerdRndfdSdTerdndfdUerdVndfdWerdndfdXerd5ndfdYerdndfdZd[erd\ndfd]erd^ndfd_d`erdndfdaerd5ndfdberdcndfdderdendfdfdgerdhndfdierdjndfdkerdjndfdlerdjndfdmerdjndfdnerdjndfdoerdjndfdperdndfdqerdrndfdserdrndfdterdrndfduerdrndfdverdrndfdwerdrndfdxerdrndfdyerdrndfdzerdrndfd{erd|ndfd}erd5ndfd~erd5ndfderd5ndfderd\ndfdderd5ndfddddderdndfderdndfderdndfddderdndfderdndfderd5ndfderd5ndfderdndfderd5ndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderd"ndfderd"ndfderdndfdderdndfderdndfderd\ndfderd\ndfderdndfderdndfderdndfderdndfdderdndfderdndfdde rdn erdrndfde rdn erdrndfde rdn erdrndfde rdn erdrndfderdndfderdndfderdndfderdndfderd\ndfdȑderdndfderdndfderdndfderdndfdΑderdndfderdndfderdndfderdndfderdndfderd\ndfderd\ndfderd5ndfderd\ndfderdndfderdndfderdndfderd5ndfderdndfderdndfderdndfderd.ndfderd.ndfdderd5ndfdderdndfde rdn erdrndfderdndfdderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfderdndfdderdndfderdjndfderdndfderdndfderdndfdddderdndfderd\ndfderdndfderdndfd erd ndfd erd ndfd erdjndfd erdndfderdndfderdndfderdndfdderdndfderd\ndfderd5ndfderdndfderdndfderdjndfdderdndfderdndfderd ndfd!erdndfd"d#d$erdndfd%erdndfd&d'e rdn erdrndfd(d)d*d+d,erd-ndfd.erdndfd/erd0ndfd1d2erdndfd3erdndfd4erd5ndfd6erd\ndfd7erdndfd8erdndfZ2ee e2Z3e jhDcic]\}}|| c}}Z5d9Z6d:Z7d;e.de.ejre.zd?e.ejre.zdzd@e:dAe-e.e.fdzdBe:e.zdzdCe.dzdDe:dEe.dBloomTokenizerBloomTokenizerFastr)rrr ministralrrr8rz.tokenization_mistral_common transformers.ztransformers.models__name__)rrrTOKENIZER_MAPPING_NAMESitemsr importlib import_modulegetattrAttributeErrorTOKENIZER_MAPPING_extra_contentvalueshasattr)r_ module_nametokenizer_classmodule tokenizer main_modules rRtokenizer_class_from_namerw\sC==  ,,&z2211+J77((  )@(E(E(G $ _ j (3K@Krr"88"001OQ_`"001[M1BDYZ vz22 '55<<> 9j$ /: = )).9K{J'{J// "  s D DDpretrained_model_name_or_path cache_dirforce_downloadproxiestokenrevisionlocal_files_only subfolderc "|jd} t|t|||||||ddd|  } | tj diSt | | } t | d5} tj| } ddd|  d<| S#1swYxYw)aY Loads the tokenizer configuration from a pretrained model tokenizer configuration. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. subfolder (`str`, *optional*, defaults to `""`): In case the tokenizer config is located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Passing `token=True` is required when you want to use a private model. Returns: `dict`: The configuration of the tokenizer. Examples: ```python # Download configuration from huggingface.co and cache. tokenizer_config = get_tokenizer_config("google-bert/bert-base-uncased") # This model does not have a tokenizer config so the result will be an empty dict. tokenizer_config = get_tokenizer_config("FacebookAI/xlm-roberta-base") # Save a pretrained tokenizer locally and you can reload its config from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased") tokenizer.save_pretrained("tokenizer-test") tokenizer_config = get_tokenizer_config("tokenizer-test") ``` _commit_hashF) ryrzr{r|r}r~r _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errorsrNz\Could not locate the tokenizer configuration file, will try to use the model config instead.rJrK) getrr loggerinfor rMrNrO) rxryrzr{r|r}r~rkwargs commit_hashresolved_config_filerQresults rRget_tokenizer_configrsJ**^,K&%%))..305 # rs %&:KHK "W 5#6"#(F> M##s BBc\eZdZdZdZeeedee zfdZ e ddZ y) AutoTokenizera This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the [`AutoTokenizer.from_pretrained`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). ctd)Nz}AutoTokenizer is designed to be instantiated using the `AutoTokenizer.from_pretrained(pretrained_model_name_or_path)` method.)OSError)selfs rR__init__zAutoTokenizer.__init__s _  r`c  |jdd}d|d<|jdd}|jdd}|jdd}|jd}|vtj|d} | ,td |d d j d tDd t | } | td| d| j |g|i|S|r3t||fi|} t| dd} tjd'i| }n| tj |fd|i|}|j} t|fi|}|jdd}d}d|vr4t|dt t"fr|d}n|djdd}|\|Z| X| dk7rStj| dj%dd|j%ddk7r t'j |g|i|Sd|vr|d|d<|r|j%dd}|du}t)|t*vxs%|duxrt |duxst |dzdu}|r:|d|d}n|d}d|vr|j-dd}nd}t/|||||}|rI|rGt1|fi|} |jdd}| j3| j |g|d|i|S|c|}t |} | |j5dst |dz} | | j6dk(rt&} | t&} | j |g|i|St9|ddrC|j:}d|vr|j%dd}t |} | j |g|i|St|t<rzt)|j>t)|j@urDtBjEd|j@jFd|j>jFd |j@}tIt)|j6xs t9|d!d}|;t*jt)|t&} | | j |g|i|S|jdd}|o|d"k7r d|vr|dd#}t |} | |j5dst |dz} | | j6dk(rt&} | t&} | j |g|i|Std$|jFd%d j d&t*Dd #t$rtj |fi|}Y)wxYw#t$r"t |j |g|i|cYSwxYw)(a Instantiate one of the tokenizer classes of the library from a pretrained model vocabulary. The tokenizer class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options Params: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. - A path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (like Bert or XLNet), e.g.: `./my_model_directory/vocab.txt`. (Not applicable to all derived classes) inputs (additional positional arguments, *optional*): Will be passed along to the Tokenizer `__init__()` method. config ([`PreTrainedConfig`], *optional*) The configuration object used to determine the tokenizer class to instantiate. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. subfolder (`str`, *optional*): In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here. tokenizer_type (`str`, *optional*): Tokenizer type to be loaded. backend (`str`, *optional*, defaults to `"tokenizers"`): Backend to use for tokenization. Valid options are: - `"tokenizers"`: Use the HuggingFace tokenizers library backend (default) - `"sentencepiece"`: Use the SentencePiece backend trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (additional keyword arguments, *optional*): Will be passed to the Tokenizer `__init__()` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__()` for more details. Examples: ```python >>> from transformers import AutoTokenizer >>> # Download vocabulary from huggingface.co and cache. >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> # Download vocabulary from huggingface.co (user-uploaded) and cache. >>> tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased") >>> # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*) >>> # tokenizer = AutoTokenizer.from_pretrained("./test/bert_saved_model/") >>> # Download vocabulary from huggingface.co and define model-specific arguments >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base", add_prefix_space=True) >>> # Explicitly use the tokenizers backend >>> tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer", backend="tokenizers") >>> # Explicitly use the sentencepiece backend >>> tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer", backend="sentencepiece") ```configNT _from_autouse_fasttokenizer_typetrust_remote_code gguf_filezPassed `tokenizer_type` z3 does not exist. `tokenizer_type` should be one of z, c3 K|]}|ywN.0cs rR z0AutoTokenizer.from_pretrained..Xs Dq Ds rfzTokenizer class z is not currently imported.F)return_tensorsrsauto_maprFastrrrz-- code_revision PythonBackendPreTrainedTokenizerFastz The encoder model config class: z3 is different from the decoder model config class: z. It is not recommended to use the `AutoTokenizer.from_pretrained()` method in this case. Please use the encoder and decoder specific tokenizer classes. model_typerz!Unrecognized configuration class z8 to build an AutoTokenizer. Model type should be one of c34K|]}|jywr)rgrs rRrz0AutoTokenizer.from_pretrained..s4[AQZZ4[sr)%poprrh ValueErrorjoinrwfrom_pretrainedrr r for_model Exceptionrrr isinstancerYlistreplacertypernrZr rregister_for_auto_classendswithrgrlrsrdecoderencoderrwarning __class__r)clsrxinputsrr_rrrtokenizer_class_namers gguf_path config_dictconfig_model_typetokenizer_configtokenizer_config_classtokenizer_auto_maphas_remote_codehas_local_code class_ref upstream_repotokenizer_class_candidate_classrs rRrzAutoTokenizer.from_pretrainedsdHd+#| JJz4 ($4d;"JJ':DAJJ{+   %#:#>#>~t#T #+ .~.>>qyy D,C DDEQH 88LMO& #34H3IId!eff2?223PdSYd]cd d #$A9WPVWI.yOPXYK))8K8F ^ c#331EVZ` #--00MXQWX!1!5!56G!N" ) )*:6 F%5j%A"%5j%A%E%EoW[%\"  &&2!-!R''++,=rBJJ6SUV%--fb9: (889VjY_jcijj - -%5n%EF> " !%;%C%CFB%O ",D8f):: "$ . )*@AMZ,-Cf-LMUYY  !!$0.q1 .q1 y ) 5a 8 $ 9!#@.Racp!  0;IGdohnoO ?D1A  3 3 52?22-06J[_e $ /(> %78QRO&/H/Q/QRX/Y";?22 ^^F/V 0E0EFm'RXZfhlJm  !/33DLBSTO*6667ThW]haghh"2!5!56G!N ! -%)<<KaAa)?)D&78NOO&/E/N/Nv/V"; X01GHXX14:>D s$T.U.UU(U?>U?Nc|||}n||}n td|||fD]}||t|j<|||t|j<tj |||y)a Register a new tokenizer in this mapping. Args: config_class ([`PreTrainedConfig`]): The configuration corresponding to the model to register. tokenizer_class: The tokenizer class to register (V5 - preferred parameter). slow_tokenizer_class: (Deprecated) The slow tokenizer to register. fast_tokenizer_class: (Deprecated) The fast tokenizer to register. Nz$You need to pass a `tokenizer_class`)exist_ok)rrrgrrnregister) config_classrsslow_tokenizer_classfast_tokenizer_classr candidates rRrzAutoTokenizer.registers  "#/"6%1"6 !GHH.0DoV MI$CL,Y-?-?@ M +0D0PEY #$8$A$A B""<8"Tr)NNNF) rg __module__ __qualname____doc__r classmethodrrhrrr staticmethodrrrrRrrsZ &'>?p 1 1p @p dkpUUrrrn)NFNNNFr)>rrjrNos collectionsrtypingrtransformers.utils.import_utilsrconfiguration_utilsrdynamic_module_utilsrr modeling_gguf_pytorch_utilsr tokenization_utils_baser utilsr r rrr utils.hubrencoder_decoderr auto_factoryrconfiguration_autorrrrrtokenization_utils_tokenizersr tokenization_utils_sentencepiecer get_loggerrgrrdictstrr__annotations__rrhrnriCONFIG_TO_TYPErSr^rwPathLikeboolrr__all__)kvs00rRrsZ #G3\?<%2*BH   H %68d3S >2702c49n-26+c3:o6C %<%>/DIC (?(A$tLC %<%>/DIC /F/H+dS C ,C,E(4P C $;$=4H C '>'@#dKC *A*C&NC (C $;$=4HC 9S9U5[_`C 3C *C +B+D'$OC 2I2K.QUVC &!C" 0G0I,tT#C$ 9%C& $;$=4H'C( &=&??TJ)C* ,+C, $;$=4H-C. "/C0 .E.G*TR1C2 &3C4 ,C,E4P5C6 %7C8 $;$=4H9C: '>'@OdK;C< "=C> /F/H+dS?C@ '>'@OdKACB (?(A$tLCCD )@)B%MECF )@)B%MGCH (?(A_tLICJ "9";FKCL &MCN "OCP 3QCR .SCT $;$=4HUCV *A*C&NWCX /F/H+dSYCZ [C\ *A*CN]C^ 1H1J-PTU_C` '>'@OdKaCb $;$=4HcCd %<%>/DIeCf gCh /F/H+dSiCj !EXEZ"A`dekCl *mCn %<%>/DIoCp )@)BoMqCr )@)BoMsCt $;$=4HuCv "wCx (?(A$tLyCz &=&?"TJ{C| '>'@#dK}C~ '>'@#dKC@ ,C,E(4PACB (?(A$tLCCD -D-F)DQECF #:#<$GGCH '>'@#dKICJ (?(A$tLKCL ,C,E(4PMCN 1H1J-PTUOCP )@)B%MQCR -D-F)DQSCT -D-F)DQUCV *A*C&NWCX ,C,E(4PYCZ )C)E%4P[C\ $;$=4H]C^ +B+D$O_C` '>'@OdKaCb +B+D'$OcCd :eCf $;$=4HgCh %iCj (kCl .mCn .oCp .E.G?TRqCr (?(A_tLsCt *A*C&NuCv +wCx &yCz (?(A$tL{C| )@)B%M}C~ ,C,E4PC@ 1H1JoPTUACB )@)B%MCCD %<%>/DIECF .E.G*TRGCH ,C,E(4PICJ 0G0I,tTKCL (?(A_tLMCN 0G0I,tTOCP 0G0I,tTQCR .E.G*TRSCT "9";FUCV 0G0I,tTWCX '>'@#dKYCZ -D-F)DQ[C\ $;$==4H]C^ "_C` (?(A$tLaCb )C)E%4PcCd (?(A$tLeCf )@)B%MgCh (B(D$$OiCj ,C,E(4PkCl &=&?"TJmCn *A*C&NoCp %qCr -D-F/DQsCt 0G0I,tTuCv 'wCz *, #)@)B%  yCF *, #)@)B%  ECR *, #)@)B%  QC^ *, #)@)B%  ]Ch &@&B"MiCj 1H1JoPTUkCl 0G0I,tTmCn &=&?"TJoCp &=&?"TJqCr $sCt !8!: EuCv &=&?]TJwCx -D-FMDQyCz "9";F{C| "}C~ %<%>/DIC@ $;$=4HACB (?(A_tLCCD (?(A$tLECF /F/H+dSGCH '>'@#dKICJ (?(A$tLKCL %<%>/DIMCN (?(A$tLOCP +B+D$OQCR )@)BoMSCT /F/H+dSUCV #:#<$GWCX &=&?"TJYCZ %<%>/DI[C\ &=&??TJ]C^ *A*C&N_C` ,C,E(4PaCb ,cCd #:#<$GeCf (gCh (?(A}tLiCl *, #)@)B%  kCv (?(A$tLwCx .yCz '>'@OdK{C| &=&?"TJ}C~ -D-F)DQC@ +B+D'$OACB ,C,E(4PCCD *A*C&NECF )@)B%MGCH &=&?"TJICJ *A*C&NKCL +B+D'$OMCN /F/H+dSOCP )@)B%MQCR -D-F)DQSCT UCV %<%>/DIWCX 0G0I,tTYCZ ,C,E(4P[C\ *A*C&N]C^ )@)BoM_C` (aCb 5cCd )eCf ,C,E(4PgCh '>'@#dKiCj $;$=4HkCl *A*CNmCn 3J3L/RVWoCp 6M6O2UYZqCr -D-F)DQsCt (B(D$$OuCv *A*C&NwCx 5O5Q1W[\yCz ,F,H(dS{C| *}C~ +B+D$OC@ +B+D'$OACB *A*CNCCD 1H1J PTUECF  7 9}tDGCH (?(A$tLICJ $KCL +B+D'$OMCN #:#<$GOCP $;$=4HQCR "9";FSCT .UCV 2WCX $;$=4HYCZ +B+D$O[C\ "]C` *, #)@)B%  _Cj -kCl 2mCn 7oCp /DIuCv $;$=4HwCx yCz 1H1J-PTU{C| 4K4M0SWX}C~ &=&?"TJC@ (?(A$tLACB *A*C&NCCD &=&?"TJECEN%%9;RS#=#7#=#=#?@41a!Q$@! )#)$s)d2B)\04 %)#"]#&S)9#9]R[[%%,]]#s(^d " ] #:  ] Dj ]]] #s(^]@`U`UF  0GAs& n