UhP~dZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZdd lmZdd lmZmZmZdd lmZmZd d lmZej<eZ dZ!dZ"d"dZ#GddejHZ%dZ&GddejHZ'eGddeZ(eGdde(Z)edGdde(eZ*edGdd e(Z+gd!Z,y)#zPyTorch CTRL model.)OptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) CTRLConfigcPdtjdd|dzz|z z }||zS)Nri')torchpow)posi d_model_size angle_ratess x/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defnr 's/eiiQ!V 'DEEK  cttj|tjj |j dtj|tjj |j d|}tj |dddddf}tj|dddddf}tj||gd}|S)Ndtyperrrdim) r rarangeint64to unsqueezesincoscat)positionrr$ angle_radssinescosines pos_encodings rpositional_encodingr4,s XU[[144U;EEaH \588?II!LJ IIjADqD) *Eii 1add7+,G99eW-26L r!c tj||jdddd}|jd}|t j |z }|6|j d|j d} } ||| | z | d| fdzz }|||z}tj|d} || |z} tj| |} | | fS) Nrrr rr%gr&)rmatmulpermuteshapenpsqrtsizesoftmax) qkvmaskattention_mask head_mask matmul_qkdkscaled_attention_logitsndnsattention_weightsoutputs rscaled_dot_product_attentionrK;s Q !Q1 56I B'"''"+5 (--b13J3O3OPR3SB4R" crc(9#:T#AA!"9N"J &=2F- 9 \\+Q /F $ $$r!c<eZdZfdZdZdZ ddZxZS)MultiHeadAttentioncnt|||_||_t ||jz |_t j|||_t j|||_ t j|||_ t j|||_ t|_ yN)super__init__ num_headsrintdepthrLinearWqWkWvdenseset pruned_heads)selfrrR __class__s rrQzMultiHeadAttention.__init__Vs "(67 ))L,7))L,7))L,7YY|\: Er!c|j|jz}t|dk(ryt||j||j\}}t |j ||_t |j||_t |j||_t |j|d|_ |jt|z |_||jz|_|jj||_y)Nrrr&) rrRlenrr[rrVrWrXrYunion)r\headsattention_head_sizeindexs r prune_headszMultiHeadAttention.prune_headsds"//4>>A u:? 7t~~Obdhduduv u%TWWe4$TWWe4$TWWe4' EqA #e*4/$..@ --33E:r!cx|j|d|j|j}|jgdS)Nr%rrrr )reshaperRrTr8)r\x batch_sizes rsplit_into_headsz#MultiHeadAttention.split_into_headsus- IIj"dnndjj Ayy&&r!c x|jd} |j|}|j|}|j|}|j || }|j || }|j || }|<|d|d} } t j | |fd}t j | |fd}|durt j||f} nd} t||||||}|djgd}|d}|j| d|j}|j|}|| f}| r||fz}|S) Nrrr6r&TrOrfr%) r9rVrWrXrjrr.stackrKr8rgrrY)r\r@r?r>rA layer_pastrBrC use_cacheoutput_attentionsripast_key past_valuepresentrJscaled_attentionattnoriginal_size_attentionoutputss rforwardzMultiHeadAttention.forwardysFWWQZ GGAJ GGAJ GGAJ  ! !!Z 0  ! !!Z 0  ! !!Z 0  !#-a=*Q-jH 8Q-R0A :q/r2A  kk1a&)GG-aAt^YW!!9,,\:ay"2":"::r4K\K\"]347# 'Gr!NNNFF)__name__ __module__ __qualname__rQrdrjrw __classcell__r]s@rrMrMUs( ";"'(r!rMctjtj||tjtj||SrO)r SequentialrUReLU)rdffs rpoint_wise_feed_forward_networkrs2 ==<5rwwy"))CQ]B^ __r!c*eZdZdfd Z ddZxZS) EncoderLayerc>t|t|||_t |||_t j|d|_t j|d|_ t j||_ t j||_ y)Ngư>eps) rPrQrMmulti_head_attentionrffnr LayerNorm layernorm1 layernorm2Dropoutdropout1dropout2)r\rrRrrater]s rrQzEncoderLayer.__init__so $6|Y$O!2<E,,|>,,|> 4(  4( r!c |j|}|j||||||||| } | d} |j| } || z} |j| } |j | } |j | } | | z} | f| ddz}|S)NrmrBrCrnrorr)rrrrrr)r\rhrArmrBrCrnronormed attn_outputs attn_outputout1out2 ffn_outputrvs rrwzEncoderLayer.forwards#00    !)/1  #1o mmK0 ;t$XXd^ ]]:. j 'L,,r!)g?rx)ryrzr{rQrwr|r}s@rrrs )qvr!rceZdZeZdZdZy)CTRLPreTrainedModel transformerct|tjtfrm|jj j d|jj|j%|jj jyyt|tjrz|jj j d|jj|j2|jj |jjyyt|tjrJ|jj j|jj jdyy)zInitialize the weights.g)meanstdN?) isinstancerrUrweightdatanormal_configinitializer_rangebiaszero_ Embedding padding_idxrfill_)r\modules r _init_weightsz!CTRLPreTrainedModel._init_weightss  fryy&1 2 MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r!N)ryrzr{r config_classbase_model_prefixrr!rrrsL%*r!rceZdZfdZdZdZdZe ddee jdee e e jdee jdee jd ee jd ee jd ee jd ee d ee dee dee dee e jeffdZxZS) CTRLModelc t|||j|_|j|_t |j|jtj|_ tj|j|j|_tj|j |_tj$t'|jDcgc]8}t)|j|j*|j,|j.:c}|_tj2|j|j4|_|j9ycc}w)Nr)rPrQn_embdrn_layer num_layersr4 n_positionsrfloatr3rr vocab_sizewr embd_pdropdropout ModuleListrangern_headr resid_pdrophrlayer_norm_epsilon layernorm post_init)r\r_r]s rrQzCTRLModel.__init__s  "MM ../0B0BDDUDUW\WbWbcf//?zz&"3"34 afgmguguav w\]\&-- FDVDV W w fmm9R9RS  xs =E*c|jSrOrr\s rget_input_embeddingszCTRLModel.get_input_embeddingss vv r!c||_yrOrr\new_embeddingss rset_input_embeddingszCTRLModel.set_input_embeddingss r!c|jD]-\}}|j|jj|/y)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrrd)r\heads_to_prunelayerras r _prune_headszCTRLModel._prune_headss>+002 BLE5 FF5M . . : :5 A Br! input_idspast_key_valuesrBtoken_type_ids position_idsrC inputs_embedsrnrooutput_hidden_states return_dictreturnc | | n|jj} ||n|jj}| | n|jj} | | n|jj} | | t d|G|j |||j} |jd| d}|jd}n0|#|jdd} |jd}n t d| |jn |j}|%d}tdgt|jz}n|ddjd}|>tj|| d|ztj |}|j#d}||dkr t d|j|d}|j#d j#d }|j%|j& }d |z tj(|j&j*z}|j-||jj.}|I|jd| d}|j1|}|t3j4|j6z}nd}||j1|}| d}tj8tj:||z||zd j%|}|t3j4|j6z}|j<j%||_|j<|ddf}||z|z}|j?|}|rd nd}| rd nd}| rd nd}tAtC|j|D]@\}\}}| r||fz}||||||||| }|dd \}}|dur||fz}| s8||d fz }B|jE|}| r||fz}| std||||fDStG||||S)aE input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer%rz5You have to specify either input_ids or inputs_embedsr6)r$devicez$batch_size has to be defined and > 0rrr#rrrTc3&K|] }|| ywrOr).0r@s r z$CTRLModel.forward..srqdedqrs)last_hidden_stater hidden_states attentions)$rrornruse_return_dict ValueError%warn_if_padding_and_no_attention_maskr<viewr9rtupler_rrr(longr+r*r$finfomin get_head_maskrrr:r;rtriuonesr3r enumerateziprr )r\rrrBrrrCrrnrorrkwargs input_shaperir past_lengthtoken_type_embedsseq_lenrA pos_embedsrpresentsall_hidden_statesall_attentionsrrrmrvrrs rrwzCTRLModel.forward s9`2C1N-TXT_T_TqTq!*!6IDKKcd d  "  6 6y. Q#..*K!r;r?;I"+J  &',,.s3K&,,Q/JTU U%.%:!! @T@T  "K#TFS[$89O)!,Q/44R8K   << [_{5RZ_ZdZdmstL'11!4L  %Q !GHH+00R@N ,55a8BB1EN,..TZZ.@N!N2ekk$**6M6Q6QQN&&y$++2E2EF  %+00[_EN $~ 6  ):):!; ;  !    FF9-Mb/zz%**W{%:GkOQ_$`rr r&+$+%   r!) NNNNNNNNNNN)ryrzr{rQrrrrrr LongTensorr FloatTensorboolrTensorr rwr|r}s@rrrsN& B15EI6:59371559$(,0/3&*^ E,,-^ "%e.?.?(@"AB^ !!2!23 ^ !!1!12 ^ u//0 ^ E--.^   1 12^ D>^ $D>^ 'tn^ d^^  uU\\"$;; <^ ^ r!rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc"eZdZdgZfdZdZdZe ddee jdee e e jdee jdee jd ee jd ee jd ee jd ee jd ee dee dee dee dee e jeffdZddZede e e jde jde e e jfdZxZS)CTRLLMHeadModelzlm_head.weightct||t||_t j |j |jd|_|jy)NTr) rPrQrrrrUrrlm_headrr\rr]s rrQzCTRLLMHeadModel.__init__sG  $V,yy0A0AM  r!c|jSrOrrs rget_output_embeddingsz%CTRLLMHeadModel.get_output_embeddingss ||r!c||_yrOrrs rset_output_embeddingsz%CTRLLMHeadModel.set_output_embeddingss % r!rrrBrrrCrlabelsrnrorrrc || | n|jj} |j|||||||| | | |  }|d}|j|}d}|*|j||fd|jj i| }| s|f|ddz}||f|zS|St |||j|j|jS)a  input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```N rrBrrrCrrnrorrrrr)losslogitsrrr) rrrr loss_functionrr rrr)r\rrrBrrrCrrrnrorrrtransformer_outputsr lm_logitsr rJs rrwzCTRLLMHeadModel.forwards v&1%..Bs.jQ[*))!X[[9J9J-KLjs58Nr)rrmrs rrz1CTRLLMHeadModel._reorder_cache..As%  j_ij j s!r)rrs `r_reorder_cachezCTRLLMHeadModel._reorder_cache8s -   r! NNNNNNNNNNNNNN)ryrzr{_tied_weights_keysrQrrrrrrrrrrrr rwr staticmethodrr|r}s@rrrs++&15EI6:59371559-1$(,0/3&*a E,,-a "%e.?.?(@"ABa !!2!23 a !!1!12 a u//0 a E--.a   1 12a ))*a D>a $D>a 'tna d^a  uU\\"$:: ;a a Fd$  uU\\23  ?D||  uU\\" #    r!ra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZfdZe ddeejdeeeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d ee de eeje ffdZxZS)CTRLForSequenceClassificationct|||j|_t||_t j |j|jd|_|jy)NFr) rPrQ num_labelsrrrrUr classifierrrs rrQz&CTRLForSequenceClassification.__init__SsR   ++$V,))FMM4??O r!rrrBrrrCrrrnrorrrc | | n|jj} |j|||||||| | | |  } | d}|j|}||jdd\}}n|jdd\}}|jj |dk7r t d|jj d}n|||jj k7j|jtj}tj|jd|jtj}||zjd}n.d}tj|jj d |tj||j |f}d}||jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}| s|f| ddz}||f|zS|St7||| j8| j:S)a2 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```Nr rrrz=Cannot handle batch sizes > 1 if no padding token is defined.r%)rr$z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)r r rr)rrrr#r9 pad_token_idrr*rrint32r(argmaxlogger warning_oncer]ry problem_typer"r$rrSr squeezerrrrrr)r\rrrBrrrCrrrnrorrrrr risequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsr loss_fctrJs rrwz%CTRLForSequenceClassification.forward\sP&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E' -;;*55   r!r)ryrzr{rQrrrrrrrrrrrwr|r}s@rr r GsW15EI6:59371559-1$(,0/3&*p E,,-p "%e.?.?(@"ABp !!2!23 p !!1!12 p u//0 p E--.p   1 12p ))*p D>p $D>p 'tnp d^p  uU\\"$<< =p p r!r )r rrrr)-__doc__typingrrrnumpyr:rrtorch.nnrrr generationr modeling_outputsr r rmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_ctrlr get_loggerryr+r r4rKModulerMrrrrrr __all__rr!rrBs! )) AA)ii-YY+   H % %4LL^`&299&R*/***@ #@ @ F S )?S  S l { $7{  { | cr!