Uh7ddlZddlmZddlmZmZmZmZmZm Z ddl Z ddl m Z mZddlmZddlmZerddlmZej*eZd d iZd Zed GddeZdgZy)N)copyfile) TYPE_CHECKINGAnyDictListOptionalTuple) AddedTokenPreTrainedTokenizer)logging)requires) TextInput vocab_fileztokenizer.modelu▁) sentencepiece)backendsc `eZdZdZeZddgZ ddeee e fffd Z dZ dZ edZd Zd d d ee ffd ZdZdZdZdZddee d ee fdZddZ d deedeeeded eeffd Z ddeedeeed eefdZ d!deededed e fdZxZS)"GemmaTokenizera Construct a Gemma tokenizer. Based on byte-level Byte-Pair-Encoding. The default padding token is unset as there is no padding token in the original model. Args: vocab_file (`str`): Path to the vocabulary file. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. sp_model_kwargs (`Dict[str, Any]`, `Optional`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. use_default_system_prompt (`bool`, *optional*, defaults to `False`): Whether or not the default system prompt for Gemma should be used. spaces_between_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add spaces between special tokens. input_idsattention_masksp_model_kwargsc  |in||_t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}||_||_||_| |_tjdi|j|_ |jj|t |4d|||||||| | | d | y)NFT) normalizedspecial) bos_token eos_token unk_token pad_token add_bos_token add_eos_tokenrclean_up_tokenization_spacesuse_default_system_promptspaces_between_special_tokens)r isinstancestrr rrr r"spmSentencePieceProcessorsp_modelLoadsuper__init__)selfrrrrrrrr r!r"r#kwargs __class__s ~/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/gemma/tokenization_gemma.pyr,zGemmaTokenizer.__init__^s &5%> r1filename_prefixctjj|stj d|dytjj ||r|dzndt dz}tjj|jtjj|k7rBtjj|jrt|j||fStjj|jsCt|d5}|jj}|j|ddd|fS|fS#1swY|fSxYw)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. zVocabulary path (z) should be a directoryN-rYrwb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathrisfileropenr)r6write)r-save_directoryraout_vocab_fileficontent_spiece_models r0save_vocabularyzGemmaTokenizer.save_vocabularys'ww}}^, LL,^,<zzzNTFFFFrv)NF)FF) __name__ __module__ __qualname____doc__rkvocab_files_namesmodel_input_namesrrr&rr,r8r=propertyr@rGrrKrOrSrWr`r rtr}intboolrrr __classcell__)r/s@r0rr+sw,\*$&67 48%*"'&+( "$sCx.1( T C .. 0[0tCy080 !x}!X]^aXb!6 sx# 9# 3;DI3F# ko# c# LJN93;DI3F cH%*.3 898"8(, 8 8r1r)reshutilrtypingrrrrrr rr'tokenization_utilsr r utilsr utils.import_utilsrtokenization_utils_baser get_loggerrrhrkrr__all__r$r1r0rs{, BBA*4   H %!#45 %&`8(`8'`8F  r1