Uhw|ddlZddlZddlZddlZddlZddlZddlmZmZddl m Z m Z ddl m Z ddlmZddlZddlmZmZddlmZej,d k7rddlZerdd lmZdd lmZdd lmZerddlZdd lm Z m!Z!m"Z"m#Z#m$Z$e%ejLejNzZ(e%ejLejRzejNzdzZ*ddiddiddiddiddiddidZ+dZ,ddj[e+j]dZ/gdZ0GddZ1e GddZ2d efd!Z3Gd"d#eZ4y)$N)ArgumentParser Namespace) dataclassfield)Thread)Optional)is_rich_availableis_torch_available)BaseTransformersCLICommandWindows)Console)Live)Markdown)AutoModelForCausalLM AutoTokenizerBitsAndBytesConfigGenerationConfigTextIteratorStreamerz .!\"#$%&'()*+,\-/:<=>?@[]^_`{|}~textz5There is a Llama in my lawn, how can I get rid of it?zyWrite a Python function that integrates any Python function f(x) numerically over an arbitrary interval [x_start, x_end].z4How many helicopters can a human eat in one sitting?z4Count to 10 but skip every number ending with an 'e'zWhy aren't birds real?z2Why is it important to eat socks after meditating?)llamacode helicopternumbersbirdssocksaa **TRANSFORMERS CHAT INTERFACE** Chat interface to try out a model. Besides chatting with the model, here are some basic commands: - **!help**: shows all available commands - **!status**: shows the current status of the model and generation settings - **!clear**: clears the current conversation and starts a new one - **!exit**: closes the interface am **TRANSFORMERS CHAT INTERFACE HELP** Full command list: - **!help**: shows this help message - **!clear**: clears the current conversation and starts a new one - **!status**: shows the current status of the model and generation settings - **!example {NAME}**: loads example named `{NAME}` from the config and uses it as the user input. Available example names: `z`, `a%` - **!set {ARG_1}={VALUE_1} {ARG_2}={VALUE_2}** ...: changes the system prompt or generation settings (multiple settings are separated by a space). Accepts the same flags and format as the `generate_flags` CLI argument. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options - **!save {SAVE_NAME} (optional)**: saves the current chat and settings to file by default to `./chat_history/{MODEL_NAME}/chat_{DATETIME}.yaml` or `{SAVE_NAME}` if provided - **!exit**: closes the interface ) )max_new_tokensr) do_sampleTr) num_beamsr r ) temperature?r!)top_k2r#)top_pr"r%)repetition_penaltyr"r&) eos_tokensN eos_token_id) eos_token_idsNr(ceZdZddeedeefdZdedefdZdefdZd Z d efd Z d ed efd Z dde fdZ dededefdZy) RichInterfaceN model_name user_namecft|_|d|_n||_|d|_y||_y)N assistantuser)r_consoler,r-)selfr,r-s l/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/commands/chat.py__init__zRichInterface.__init__xs3   )DO(DO  #DN&DN output_streamreturncXd}|jjd|jdt|jd5}t |D]\}}|r|dk(r||z }g}|j D]G}|j ||jdr|j d7|j d Itdj|jd }|j| d d d |jj|S#1swY%xYw) zRStream output from a role, and return the generated text after it's done steaming.z [bold blue]:)consolerefresh_per_secondrz``` z z github-dark) code_themeN) r1printr,r enumerate splitlinesappend startswithrjoinstripupdate) r2r6rliveioutputslineslinemarkdowns r3 stream_outputzRichInterface.stream_outputs   l4??*;2>? $--A > &$' 6 & 7!q& OO--DLL&u- T* V,-$BGGEN$8$8$:}U H%5 & &:  = & &s B8D  D)c|jjd|jd}|jj|S)z!Gets user input from the console. [bold red]: )r1inputr-r?)r2rPs r3rPzRichInterface.inputs9 ##k$..1A$FG  r5c8|jjy)zClears the console.N)r1clear)r2s r3rRzRichInterface.clears r5rc|jjd|jd||jjy)z%Prints a user message to the console.rOz>:[/ bold red] N)r1r?r-)r2rs r3print_user_messagez RichInterface.print_user_messages7 k$..)99I$PQ r5colorcz|jjd|d||jjy)z,Prints text in a given color to the console.z[bold ]Nr1r?)r2rrUs r3 print_colorzRichInterface.print_colors1 fUG1TF34 r5minimalc|jjt|rtnt|jjy)z'Prints the help message to the console.N)r1r?rHELP_STRING_MINIMAL HELP_STRING)r2rZs r3 print_helpzRichInterface.print_helps. HG%8UV r5generation_config model_kwargsc|jjd|d|r|jjd||jjd||jjy)zFPrints the status of the model and generation settings to the console.z[bold blue]Model: r=z[bold blue]Model kwargs: z [bold blue]NrX)r2r,r_r`s r3 print_statuszRichInterface.print_statussf 0 B?@  MM  ";L> J K k*;)<=> r5)NN)F)__name__ __module__ __qualname__rstrr4rrMrPrRrTrYboolr^rdictrbr5r3r+r+ws '8C= 'HSM '%+?%C%Ns s C $ s?O_cr5r+ceZdZUdZedddiZeeed<edddiZ eeed<eddd iZ eeed <ed dd iZ eed <edddiZ eeed<edddiZ eeed<edddiZeed<edddiZeed<edddiZeed<edddiZeed<edddiZeed <eddd!iZeed"<eddd#iZeed$<eddd%iZeeed&<eddd'iZeeed(<ed)dd*iZeed+<ed,dd-iZeed.<ed/d0gd1d2Zeeed3<ed4dd5iZeed6<eddd7iZeeed8<ed4dd9iZeed:<ed4dd;iZ eed<<ed=d>d?d=gd2Z!eed@<ed4ddAiZ"eedB<y)C ChatArgumentsz Arguments for the chat CLI. See the metadata arg for each argument's description -- the medatata will be printed with `transformers chat --help` Nhelpz_Name of the pre-trained model. The positional argument will take precedence if both are passed.)defaultmetadatamodel_name_or_pathzKUsername to display in chat interface. Defaults to the current user's name.r0zSystem prompt. system_promptz./chat_history/zFolder to save chat history. save_folderz"Path to a yaml file with examples. examples_pathzPath to a local generation config file or to a HuggingFace repo containing a `generation_config.json` file. Other generation settings passed as CLI arguments will be applied on top of this generation config.r_rz%Maximum number of tokens to generate.rTz,Whether to sample outputs during generation.rr z Number of beams for beam search.r r"z%Temperature parameter for generation.r!r$zValue of k for top-k sampling.r#z Value of p for nucleus sampling.r%zRepetition penalty.r&zNEOS tokens to stop the generation. If multiple they should be comma separated.r'zQEOS token IDs to stop the generation. If multiple they should be comma separated.r)mainzLSpecific model version to use (can be a branch name, tag name or commit id).model_revisioncpuzDevice to use for inference.deviceautozOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.)rwbfloat16float16float32)rlchoices torch_dtypeFz2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant)#rcrdre__doc__rrorrf__annotations__r0rprqrrr_rintrrgr r!floatr#r%r&r'r)rtrvr|r}r~rrrrrir5r3rkrks). u )   ghD(3-$)IY@Z#[M8C=[%6&JhAijKj#(Im@n#oM8C=o(- 1  (x}  v?f6ghNChDFe5fgKgrV5U,VWE3W8Z/[\E5\ %cVEZ<[ \\ %jk!J $)mn$M8C= hiNC9W0XYFCY!&PA "K#$)] ^t*/ r *#efL$efL$ %UFZhmotgu=vww!&uHm?n!o$or5rkargsct|S)z; Factory function used to chat with a local model. ) ChatCommand)rs r3chat_command_factoryr%s t r5ceZdZedefdZdZdedefdZede fdZ ed ded e e de fd Z ed d e e de efd Zd e e defdZdededeeeffdZedede e de e deee effdZedede dfdZdedeeeffdZde dededee ee e ffdedede edee eeeffdZdZy)!rparserctf}|jd|}|jd}|jdtdd|jdtdd d |j t y) z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments chat)dataclass_typeszPositional argumentsmodel_name_or_path_positionalNzName of the pre-trained model.)typermrlgenerate_flagsaFlags to pass to `generate`, using a space as a separator between flags. Accepts booleans, numbers, and lists of integers, more advanced parameterization should be set through --generation-config. Example: `transformers chat max_new_tokens=100 do_sample=False eos_token_id=[1,2]`. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options*)rrmrlnargs)func)rk add_parseradd_argument_group add_argumentrf set_defaultsr)rr chat_parsergroups r3register_subcommandzChatCommand.register_subcommand-s)*'''P ../EF  +#tJj    J      &: ;r5c4|j|}||_yN)_handle_deprecated_argsr)r2rs r3r4zChatCommand.__init__Js++D1 r5rr7c d}|jxs |j|_|j td|jd}tjdt t D]>\}}}t||}||k7sd}tjd|d|d|dt @|rtd t|S) z Handles deprecated arguments and their deprecation cycle. To be removed after we fully migrated to the new args. FzOne of the following must be provided: - The positional argument containing the model repo, e.g. `transformers chat ` - the optional --model_name_or_path argument, containing the model repo (deprecated)TzThe --model_name_or_path argument is deprecated will be removed in v4.54.0. Use the positional argument instead, e.g. `transformers chat `.zThe --a. argument is deprecated will be removed in v4.54.0. There are two alternative solutions to specify this generation option: 1. Pass `--generation-config ` to specify a generation config. 2. Pass `generate` flags through positional arguments, e.g. `transformers chat =`z (Press enter to continue)) rro ValueErrorwarningswarn FutureWarning_DEPRECATION_MAPgetattrr?rP)r2r has_warningsdeprecated_arg default_valuenew_argvalues r3rz#ChatCommand._handle_deprecated_argsNs  .2-O-O-jSWSjSj*  - - 5i   $ $ 0L MMK  7G  2NM7D.1E %#  ^,-siqq * "    / 0 G r5ctjdk(rtjSt j tj jS)z)Returns the username of the current user.r )platformsystemosgetloginpwdgetpwuidgetuidpw_namerir5r3 get_usernamezChatCommand.get_usernameys: ??  );;= << ,44 4r5Nfilenameci}t||d<||d<|j}|Gtjd}|jd|d}t j j||}t jt j j|dt|d 5}tj||d dddt j j|S#1swY(xYw) z!Saves the chat history to a file.settings chat_historyNz%Y-%m-%d_%H-%M-%Sz/chat_.jsonT)exist_okwr:)indent)varsrqtimestrftimerrpathrDmakedirsdirnameopenjsondumpabspath)rrr output_dictfoldertime_strfs r3 save_chatzChatCommand.save_chats "&t* J&* N#!!  }}%89H<<=VH:USHww||FH5H BGGOOH-= (C  0A IIk1Q / 0wwx(( 0 0s +C++C4rpc|g}|Sd|dg}|S)zClears the chat history.rrolecontentri)rprs r3clear_chat_historyzChatCommand.clear_chat_historys*  D &-@AD r5rc t|dk(riS|Dcic]/}d|jddzdz|jdd1}}|jDcic]*\}}||jdvr|jn|,}}}|jDcic]\}}||dk(rdn|}}}dtd t fd }|jDcic]\}}|||sd|dn|}}}d j |jDcgc] \}}|d |c}}}d |zdz}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd} tj|}|Scc}wcc}}wcc}}wcc}}wcc}}w#tj$r tdwxYw)zUParses the generate flags from the user input into a dictionary of `generate` kwargs.r"rr )truefalseNonenullsr7cD|jdddjS)N.r9r )replaceisdigit)rs r3 is_numberz3ChatCommand.parse_generate_flags..is_numbers99S"a(002 2r5z, z: {}z"null"z"true"rz"false"rz"[[z]"rW:zFailed to convert `generate_flags` into a valid JSON object. `generate_flags` = {generate_flags} Converted JSON string = {generate_flags_string}) lensplititemslowerrfrgrDrrloadsJSONDecodeErrorr) r2rflaggenerate_flags_as_dictkvrgenerate_flags_stringprocessed_generate_flagss r3parse_generate_flagsz ChatCommand.parse_generate_flagss@ ~ ! #I bp!pY]# 3(:":S"@$**S/RSBT"T!p!p OeNjNjNl" FJaAAGGI)::qwwy A" " OeNjNjNl!mda!qF{V"A!m!m 3 3 3VlUqUqUs!tTQPQ!Yq\q1Xq"H!t!t !% CYC_C_Ca*b41aaS1#;*b c!$&; ;c A!6 = =h O 5 = =h O 5 = =i Q 5 = =dC H 5 = =dC H!6 = =c3 G '+zz2G'H $('S"q" "n "u +c"## D  s)4F>/G#G GG 'GG: tokenizerc |jt}|j||j|j\}}|j |j |j|j|j|j|j||d }|jdi|nd|jvritjj|j}tjj!|j}tj"||}ntj"|j}|j%|j&} |jdi| } || fS)zj Returns a GenerationConfig object holding the generation parameters for the CLI command. ) rrr r!r#r%r& pad_token_idr(rri)r_rparse_eos_tokensr'r)rrr r!r#r%r&rFrrrbasenamefrom_pretrainedrr) r2rrr_rr)deprecated_kwargsrrparsed_generate_flagsr`s r3get_generation_parameterizationz+ChatCommand.get_generation_parameterizationsB  ! ! ) 0 2 *.*?*? 4??\`\n\n*o 'L-"&"5"5!^^!^^#//&*&=&= , - !  %  $ $ 9'8 9$000''//$*@*@A77++D,B,BC$4$D$DWh$W!$4$D$DTE[E[$\!!% 9 9$:M:M N/(//H2GH !,..r5r'r)c|j |j}n |j}g}|/|j|j|j d|7|j|j dDcgc] }t |c}t |dk(r|j|j||fScc}w)z:Retrieves the pad token ID and all possible EOS token IDs.,r)rr(extendconvert_tokens_to_idsrrrrB)rr'r)rall_eos_token_idstoken_ids r3rzChatCommand.parse_eos_tokenss  ! ! )$11L$11L  !  $ $Y%D%DZEUEUVYEZ%[ \  $  $ $MDWDWX[D\%]c(m%] ^  !Q &  $ $Y%;%; <... &^s9B? model_argsrc|jr:td|j|j|j|j}|S|j rtd}|Sd}|S)NT)rbnb_4bit_compute_dtyperbnb_4bit_use_double_quantbnb_4bit_quant_storage)r)rrr|rrr)rquantization_configs r3get_quantization_configz#ChatCommand.get_quantization_configst  " ""4!'1'='=$.$B$B*4*I*I'1'='= # #" $ $"4!#  #"#' ""r5ctj|j|j|j}|j dvr |j nt t|j }|j|}|j|j|d|d}tj|jfd|ji|}t |dd|j|j}||fS)N)revisionr})rwNrw)r r~r| device_maprr} hf_device_map) rrrrtr}r|rtorchrr~rtorv)r2rrr|rr`models r3load_model_and_tokenizerz$ChatCommand.load_model_and_tokenizer&s!11  . .(("44 +/*:*:n*Ld&&RYZ_aeaqaqRr "::4@++#'#;#;& #6  %44  . . BFBXBX \h  5/4 0 8HHT[[)Eir5 user_input interfaceexamplesr_r`rc|dk(r-|j|j}|jn-|dk(r|jn|j drnt |j dkrR|j }t |dk(r|d} nd} |j||| } |jd| dd n|j d r|d dj} | j } | D] } d | vs|jd| dd n?|j| } |jdi| } |jdi| n|j drt |j dk(r|j d}||vrD|jg}|j||d|jd||ddnd|dt|jd}|j|d nK|dk(r|j!|j"||n'|jd|dd |j|||fS)z Handles all user commands except for `!exit`. May update the chat history (e.g. reset it) or the generation config (e.g. set a new flag). z!clearz!helpz!saver NzChat saved in !green)rrUz!setr:rz(Invalid flag format, missing `=` after `z;`. Please use the format `arg_1=value_1 arg_2=value_2 ...`.red!examplerr0rzExample z* not found in list of available examples: rz!status)r,r_r`'z/' is not a valid command. Showing help message.ri)rrprRr^rCrrrrYrErrFrTrBlistkeysrbr)r2rrrrr_r`r split_inputrnew_generate_flagsrparsed_new_generate_flagsnew_model_kwargs example_name example_errors r3handle_non_exit_user_commandsz)ChatCommand.handle_non_exit_user_commandsAs  !**4+=+=>D OO  7 "  "  " "7 +J4D4D4F0G!0K$**,K;1$&q>~~dD(;H  ! !z'C7 ! S  " "6 *",AB!5!5!7 !3!9!9!; * 8d?))FtfMAA$ * 8-1,E,EFX,Y)#;#4#;#;#X>W#X # ##7&67  " ": .3z7G7G7I3Ja3O%++-a0Lx'!,,Xl-CF-KL V8Nv8VWX|n,VW[\d\i\i\kWlVmmno%%=%F 9 $  " "=="3) #   ! !:,6e'fns ! t  "& 44r5c ts tdts td|j}|jt }n3t |j5}tj|}ddd|j|j}n |j}|j|\}}t|dd}|j||\}} t|j|} | j!|j#|j$} | j'd | j)} | j+dr4| dk(ry|j-| || || | \} }} | j+d sW| j/d | d |j1| d dj3|j4} t7j8| }| |||d| }t;|j<|}|j?| jA|}|jC| j/d|d #1swYxYw#tD$rYywxYw)NzHYou need to install rich to use the chat interface. (`pip install rich`)zJYou need to install torch to use the chat interface. (`pip install torch`)T)skip_special_tokens skip_prompt)r,r-)rZrz!exit)rrrrr_r`rrr0rpt)return_tensorsadd_generation_prompt)inputsattention_maskstreamerr_)targetkwargsr/)#r ImportErrorr rrrDEFAULT_EXAMPLESryaml safe_loadr0rrrrr+rrRrrpr^rPrCr#rBapply_chat_templaterrvr  ones_likergeneratestartrMrDKeyboardInterrupt)r2rrrr0rrgeneration_streamerr_r`rrrr*r+generation_kwargsthread model_outputs r3runzChatCommand.runs\ "hi i!#jk kyy    %'Hd(() -Q>>!, - 99 $$&D99D88>y29RVdhi*.*N*NtU^*_'<!T-O-O[_` &&t'9'9: T*+ &__. ((-!W,@D@b@b'1!%&/%-.?)5!%AcA=/&00< KKJ GH"66tDhl6mppLL"'!8$&4 3): % # %! u~~>OP (667JK   [\JKS% - -z%  s+I3&I-IB;II IIr)rcrdre staticmethodrrr4rkrrfrrrrrhrrrtuplerrrrrrrr+r#r<rir5r3rr,s\##& ] uEY[hEh?i 6N5N5N5! N5 sDcN*+ N5 , N5N54jN5 tDz+T1 2N5dHr5r)5rrrstringrrargparserr dataclassesrr threadingrtypingrr1transformers.utilsr r r9r rr rich.consoler rich.liver rich.markdownrr  transformersrrrrrset ascii_letters whitespaceALLOWED_KEY_CHARSdigitsALLOWED_VALUE_CHARSr0r\rDrr]rr+rkrrrir5r3rOs}   .( D(8?? !$&,,v/@/@@A 6==(6+<+<?@A & RRj UpUp Uppyo,or5