Uh`dZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z m Z mZddlmZmZdd lmZmZdd lmZdd lmZdd lmZmZmZd dlmZej>e Z!eGddeZ"eGddeZ#eGddeZ$eGddeZ%eGddeZ&eGddeZ'eGddeZ(eGddeZ)eGdd eZ*eGd!d"eZ+Gd#d$e jXZ-Gd%d&e jXZ.Gd'd(e jXZ/Gd)d*e jXZ0Gd+d,e jXZ1Gd-d.e jXZ2Gd/d0e jXZ3Gd1d2e jXZ4Gd3d4e jXZ5Gd5d6e jXZ6Gd7d8e jXZ7Gd9d:e jXZ8eGd;dGd?d@e9Z:dAZ;GdBdCe jXZ<edD>GdEdFe9Z=edG>GdHdIe9Z>edJ>GdKdLe9Z?edM>GdNdOe9Z@edP>GdQdRe9ZAedS>GdTdUe9ZBeGdVdWe9ZCeGdXdYe9ZDgdZZEy)[zPyTorch LUKE model.N) dataclass)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNgelu)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel)apply_chunking_to_forward) ModelOutputauto_docstringlogging) LukeConfigcleZdZUdZdZeejed<dZ ee ejdfed<y)BaseLukeModelOutputWithPoolinga Base class for outputs of the LUKE model. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. entity_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, entity_length, hidden_size)`): Sequence of entity hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length + entity_length, sequence_length + entity_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nentity_last_hidden_state.entity_hidden_states __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrx/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/luke/modeling_luke.pyrr%@2=Ahu'8'89@DH(5):):C)?#@AHr$rcleZdZUdZdZeejed<dZ ee ejdfed<y)BaseLukeModelOutputa# Base class for model's outputs, with potential hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. entity_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, entity_length, hidden_size)`): Sequence of entity hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr.rrr#r$r%r(r(Dr&r$r(cleZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eeejed<dZeeejd fed <dZeeejd fed <y) LukeMaskedLMOutputa> Base class for model's outputs, with potential hidden states and attentions. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): The sum of masked language modeling (MLM) loss and entity prediction loss. mlm_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Masked language modeling (MLM) loss. mep_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Masked entity prediction (MEP) loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). entity_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the entity prediction head (scores for each entity vocabulary token before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossmlm_lossmep_losslogits entity_logits hidden_states.r attentions)rrrrr+rr r!r"r,r-r.r/r0rrr1r#r$r%r*r*cs>)-D(5$$ %,,0Hhu(()0,0Hhu(()0*.FHU&& '.15M8E--.58Ju00#567>r$r*ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<y) EntityClassificationOutputay Outputs of entity classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr+r..r0rr1rrrrr+rr r!r"r.r0rrr1r#r$r%r3r3,)-D(5$$ %,*.FHU&& '.=AM8E%"3"3S"89:ADH(5):):C)?#@AH:>Ju00#567>r$r3ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<y) EntityPairClassificationOutputa~ Outputs of entity pair classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr+r..r0rr1r4r#r$r%r7r7r5r$r7ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<y) EntitySpanClassificationOutputa Outputs of entity span classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, entity_length, config.num_labels)`): Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr+r..r0rr1r4r#r$r%r9r9r5r$r9ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<y) LukeSequenceClassifierOutputa Outputs of sentence classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr+r..r0rr1r4r#r$r%r;r;2)-D(5$$ %,*.FHU&& '.=AM8E%"3"3S"89:ADH(5):):C)?#@AH:>Ju00#567>r$r;ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<y) LukeTokenClassifierOutputa Base class for outputs of token classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) : Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_labels)`): Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr+r..r0rr1r4r#r$r%r>r> r<r$r>c eZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZee ejdfed <y) LukeQuestionAnsweringModelOutputay Outputs of question answering models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Span-end scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr+ start_logits end_logits.r0rr1)rrrrr+rr r!r"rArBr0rrr1r#r$r%r@r@/s6)-D(5$$ %,04L(5,,-4.2J**+2=AM8E%"3"3S"89:ADH(5):):C)?#@AH:>Ju00#567>r$r@ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<y) LukeMultipleChoiceModelOutputa  Outputs of multiple choice models. Args: loss (`torch.FloatTensor` of shape *(1,)*, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, num_choices)`): *num_choices* is the second dimension of the input tensors. (see *input_ids* above). Classification scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr+r..r0rr1r4r#r$r%rDrDTs6)-D(5$$ %,*.FHU&& '.=AM8E%"3"3S"89:ADH(5):):C)?#@AH:>Ju00#567>r$rDc8eZdZdZfdZ ddZdZxZS)LukeEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j |_tj|j|j |j"|_ y)N padding_idxeps)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutrIselfconfig __class__s r%rMzLukeEmbeddings.__init__}s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= "..#%<<  * *F,>,>DL\L\$  r$c|C|0t||jj|j}n|j |}||j }n|j dd}|:t j|t j|jj}||j|}|j|}|j|}||z|z}|j|}|j|}|S)Ndtypedevice)"create_position_ids_from_input_idsrItord&create_position_ids_from_inputs_embedssizer zeroslong position_idsrRrTrVrWr[) r] input_idstoken_type_idsrk inputs_embeds input_shaperTrV embeddingss r%forwardzLukeEmbeddings.forwards  $A)TM]M]^aabkbrbrs #JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN   00;M"66|D $ : :> J"%88;PP ^^J/ \\*- r$c|jdd}|d}tj|jdz||jzdztj|j }|j dj|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrarrbr)rhr arangerIrjrd unsqueezeexpand)r]rnrosequence_lengthrks r%rgz5LukeEmbeddings.create_position_ids_from_inputs_embedss$((*3B/ %a.||   q /D4D4D"Dq"HPUPZPZcpcwcw %%a(// <S>STzz&"<"<= r$ entity_idsrkrmcR|tj|}|j|}|jj|jj k7r|j |}|j|jd}|dk7j|jd}||z}tj|d}||jdjdz }|j|}||z|z}|j|}|j|}|S)Nr)minradimgHz>)r zeros_likerr^rrPrrTclamptype_asrtsumrVrWr[) r]rrkrmrrTposition_embedding_maskrVrps r%rqzLukeEntityEmbeddings.forwards'  !"--j9N 22:> ;; & &$++*A*A A $ ; ;">?R"S"]"]^`"a14KK#ii(;D14K4O4OTV4O4W4]4]bf4]4gg $ : :> J&)<z >$6: $$&&!!1!12 r$rzc2eZdZfdZdZ ddZxZS)LukeSelfAttentionc t||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_|j|_ tj|j|j|_ tj|j|j|_ tj|j|j|_|jrtj|j|j|_tj|j|j|_tj|j|j|_tj$|j&|_y)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)rLrMrPnum_attention_headshasattr ValueErrorintattention_head_size all_head_sizeuse_entity_aware_attentionrrquerykeyvalue w2e_query e2w_query e2e_queryrYattention_probs_dropout_probr[r\s r%rMzLukeSelfAttention.__init__s     : : :a ?PVXhHi"6#5#5"67334A7  $*#=#= #&v'9'9FS"S* **,CD $1!ZiZ2B$C!  '*. '*79:q8H*I ' 02M_G12MNGr$NNF)rrrrMrrqrwrxs@r%rrsG0% Pr$rcneZdZfdZdej dej dej fdZxZS)LukeSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNrJ) rLrMrrrPdenserWrXrYrZr[r\s r%rMzLukeSelfOutput.__init___s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r$r0 input_tensorreturncr|j|}|j|}|j||z}|Srrr[rWr]r0rs r%rqzLukeSelfOutput.forwarde7 =1  ]3 }|'CD r$rrrrMr Tensorrqrwrxs@r%rr^1> U\\RWR^R^r$rc2eZdZfdZdZ ddZxZS) LukeAttentionct|t||_t ||_t |_yr)rLrMrr]routputset pruned_headsr\s r%rMzLukeAttention.__init__ms0 %f- $V, Er$ctdNz4LUKE does not support the pruning of attention headsNotImplementedError)r]headss r% prune_headszLukeAttention.prune_headss!"XYYr$cF|jd}|j|||||}||d}|} n3tj|ddd}tj||gd} |j || } | ddd|ddf} |d} n| dd|dddf} | | f|ddz} | S)Nrrrr)rhr]r rr)r]rrrrrr self_outputsconcat_self_outputsrattention_outputword_attention_outputentity_attention_outputrs r%rqzLukeAttention.forwardvs'++A. yy       '".q/ #5 "'))L!,S>STzz&"<"<= r$r0rrcr|j|}|j|}|j||z}|Srrrs r%rqzLukeOutput.forwardrr$rrxs@r%rrrr$rc2eZdZfdZ ddZdZxZS) LukeLayerct||j|_d|_t ||_t ||_t||_ yNr) rLrMchunk_size_feed_forward seq_len_dimr attentionr intermediaterrr\s r%rMzLukeLayer.__init__sI '-'E'E$&v.,V4 ( r$cJ|jd}|j|||||}||d}ntj|ddd}|dd} t |j |j |j|} | ddd|ddf} |d} n| dd|dddf} | | f| z} | S)Nr)rrrr)rhrr rrfeed_forward_chunkrr) r]rrrrrrself_attention_outputsconcat_attention_outputr layer_outputword_layer_outputentity_layer_outputs r%rqzLukeLayer.forwards'++A. !%   / "0"  '&z&LukeEncoder.forward..( = )last_hidden_stater0r1rr) enumeraterrtraining_gradient_checkpointing_func__call__tupler()r]rrrrroutput_hidden_states return_dictall_word_hidden_statesall_entity_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r%rqzLukeEncoder.forwardsi(<)=24 $5b4(4 POA|#)?CUBW)W&+CG[F]+](.7.CilO**t}} $ A A ))&("#% ! !-&("#% ! "/q!1 #/'4Q'7$ &9]1=MS%S "'?CWBY'Y $ '*'(,    #00*%9!9   r$)NNFFTrrrrMrqrwrxs@r%rrs,"D r$rcVeZdZfdZdej dej fdZxZS) LukePoolerct|tj|j|j|_tj |_yr)rLrMrrrPrTanh activationr\s r%rMzLukePooler.__init__>s9 YYv1163E3EF '')r$r0rc\|dddf}|j|}|j|}|S)Nr)rr7)r]r0first_token_tensor pooled_outputs r%rqzLukePooler.forwardCs6+1a40 #56  6 r$rrxs@r%r4r4=s#$ U\\ellr$r4c$eZdZfdZdZxZS)EntityPredictionHeadTransformcht|tj|j|j |_t|jtrt|j|_ n|j|_ tj|j |j|_ yr)rLrMrrrPrrrrrr transform_act_fnrWrXr\s r%rMz&EntityPredictionHeadTransform.__init__Ms{ YYv1163I3IJ f'' -$*6+<+<$=D !$*$5$5D !f&<&<&BWBWXr$cl|j|}|j|}|j|}|Sr)rr>rWrs r%rqz%EntityPredictionHeadTransform.forwardVs4 =1 --m< }5 r$r2rxs@r%r<r<LsYr$r<c$eZdZfdZdZxZS)EntityPredictionHeadc t|||_t||_t j |j|jd|_ t jtj|j|_ y)NFr|)rLrMr^r< transformrrrr~decoder Parameterr rir}r\s r%rMzEntityPredictionHead.__init__^sa  6v>yy!7!79Q9QX]^ LLV-E-E!FG r$cd|j|}|j||jz}|Sr)rCrDr}rs r%rqzEntityPredictionHead.forwardes-}5  ]3dii? r$r2rxs@r%rArA]sHr$rAc@eZdZeZdZdZddgZdejfdZ y)LukePreTrainedModellukeTrrzmodulecjt|tjrm|jjj d|j j|j%|jjjyyt|tjr|jdk(r%|jjjn;|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weightsg)meanstdNr?)rrrweightdatanormal_r^initializer_ranger}zero_rN embedding_dimrIrWfill_)r]rJs r% _init_weightsz!LukePreTrainedModel._init_weightsss( fbii ( MM   & &CT[[5R5R & S{{&   &&('  -##q( ""((* ""**9V9V*W!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r$N) rrrr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modulesrModulerVr#r$r%rHrHls/L&*#(*@A*BII*r$rHzt The bare LUKE model transformer outputting raw hidden-states for both word tokens and entities without any ) custom_introc eZdZddedeffd ZdZdZdZdZ dZ e dd e e jd e e jd e e jd e e jd e e jde e jde e jde e jde e jde e jde ede ede edeeeffdZde jde e jfdZxZS) LukeModelr^add_pooling_layerct||||_t||_t ||_t||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rLrMr^rFrprzrrencoderr4pooler post_init)r]r^r_r_s r%rMzLukeModel.__init__sZ   (0!5f!="6* ,=j(4  r$c.|jjSrrprRr]s r%get_input_embeddingszLukeModel.get_input_embeddingss...r$c&||j_yrrer]rs r%set_input_embeddingszLukeModel.set_input_embeddingss*/'r$c.|jjSrrrfs r%get_entity_embeddingszLukeModel.get_entity_embeddingss%%777r$c&||j_yrrlris r%set_entity_embeddingszLukeModel.set_entity_embeddingss380r$ctdrr)r]heads_to_prunes r% _prune_headszLukeModel._prune_headsrr$rlrrmrkrentity_attention_maskentity_token_type_idsentity_position_idsrrnrr)r*rc | | n|jj} | | n|jj} | | n|jj} | | t d|#|j |||j }n!| | j dd}n t d|\}}| |jn | j}|tj||f|}|&tj|tj|}|V|j d}|tj||f|}|(tj||ftj|}|j| |jj} |j|||| }|j||}|d}n|j!|||}|j#|||| | | | }|d }|j$|j%|nd}| s ||f|ddzSt'|||j(|j*|j,|j. S) uz entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, LukeModel >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base") >>> model = LukeModel.from_pretrained("studio-ousia/luke-base") # Compute the contextualized entity representation corresponding to the entity mention "Beyoncé" >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [(0, 7)] # character-based entity span corresponding to "Beyoncé" >>> encoding = tokenizer(text, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt") >>> outputs = model(**encoding) >>> word_last_hidden_state = outputs.last_hidden_state >>> entity_last_hidden_state = outputs.entity_last_hidden_state # Input Wikipedia entities to obtain enriched contextualized representations of word tokens >>> text = "Beyoncé lives in Los Angeles." >>> entities = [ ... "Beyoncé", ... "Los Angeles", ... ] # Wikipedia entity titles corresponding to the entity mentions "Beyoncé" and "Los Angeles" >>> entity_spans = [ ... (0, 7), ... (17, 28), ... ] # character-based entity spans corresponding to "Beyoncé" and "Los Angeles" >>> encoding = tokenizer( ... text, entities=entities, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt" ... ) >>> outputs = model(**encoding) >>> word_last_hidden_state = outputs.last_hidden_state >>> entity_last_hidden_state = outputs.entity_last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timeraz5You have to specify either input_ids or inputs_embeds)rdrbr)rlrkrmrn)rrrr)r*r)r# pooler_outputr0r1rr)r^rr)use_return_dictr%warn_if_padding_and_no_attention_maskrhrdr onesrirj get_head_maskrrpget_extended_attention_maskrrarbrr0r1rr)r]rlrrmrkrrsrtrurrnrr)r*ro batch_size seq_lengthrdentity_seq_lengthword_embedding_outputextended_attention_maskentity_embedding_outputencoder_outputssequence_outputr:s r%rqzLukeModel.forwardsR2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T  !"ZZZ(@PN  !"[[EJJvVN  ! * 2 $,(- J@Q3R[a(b%$,(- ZAR4S[`[e[ent(u%&&y$++2O2OP !%%)' !0! #'"B"B>Sh"i  &* #&*&<&1559,0/3&*Y E,,-Y !!2!23Y !!1!12 Y u//0 Y U--. Y  ((9(9:Y  ((8(89Y &e&6&67Y E--.Y   1 12Y $D>Y 'tnY d^Y  u44 5Y Y v'#(#3#3'LTUZUeUeLf'r$r^c|j|j}tj|dj ||z}|j |zS)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)nerr cumsumrrj)rlrImaskincremental_indicess r%rerefsP << $ ( ( *D <<!4<->!?@  II r$c |j|}t|}|j|}|j|}|Sr)rr rrD)r]featureskwargsrs r%rqzLukeLMHead.forwards; JJx  G OOA  LLOr$c|jjjjdk(r|j|j_y|jj|_y)Nmeta)rDr}rdtyperfs r% _tie_weightszLukeLMHead._tie_weightssC <<   # # ( (F 2 $ DLL  ))DIr$)rrrrrMrqrrwrxs@r%rrws4&*r$rz The LUKE model with a language modeling head and entity prediction head on top for masked language modeling and masked entity prediction. c$eZdZgdZfdZfdZdZdZe dde e jde e jde e jd e e jd e e jd e e jd e e jd e e jde e jde e jde e jde e jde e de e de e deeeff dZxZS)LukeForMaskedLM)zlm_head.decoder.weightzlm_head.decoder.biasz!entity_predictions.decoder.weightct||t||_t ||_t ||_tj|_ |jyr) rLrMr^rIrlm_headrAentity_predictionsrr loss_fnrcr\s r%rMzLukeForMaskedLM.__init__sQ  f% !&) "6v">**,  r$ct||j|jj|j j j yr)rL tie_weights_tie_or_clone_weightsrrDrIr)r]r_s r%rzLukeForMaskedLM.tie_weightss:  ""4#:#:#B#BDIID_D_DqDqrr$c.|jjSrrrDrfs r%get_output_embeddingsz%LukeForMaskedLM.get_output_embeddingss||###r$c&||j_yrr)r]new_embeddingss r%set_output_embeddingsz%LukeForMaskedLM.set_output_embeddingss- r$rlrrmrkrrsrtrulabels entity_labelsrrnrr)r*rcJ||n|jj}|j||||||||| | | |d }d}d}|j|j}| d| j |j } |j|jd|jj| jd}||}d}d}|jl|j|j}| O|j|jd|jj| jd}||}n||z}|s8td||||||j|j|j fDSt#||||||j|j|j S)aC entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` entity_labels (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` NT rlrrmrkrrsrtrurrnrr)r*rac3$K|]}|| ywrr#rs r%r z*LukeForMaskedLM.forward..s = r")r+r,r-r.r/r0rr1)r^rxrIrr#rfrdrrrOrrr~r(r0rr1r*)r]rlrrmrkrrsrtrurrrrnrr)r*rr+r,r.r-r/s r%rqzLukeForMaskedLM.forwardsb&1%-1481559,0/3&*!q E,,-q !!2!23q !!1!12 q u//0 q U--. q  ((8(89q  ((8(89q &e&6&67q ))*q   0 01q E--.q   1 12q $D>q 'tnq d^!q " u(( )#q q r$rz The LUKE model with a classification head on top (a linear layer on top of the hidden state of the first entity token) for entity classification tasks, such as Open Entity. c"eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e e ffdZ xZS)LukeForEntityClassificationc,t||t||_|j|_t j |j|_t j|j|j|_ |jyr rLrMr^rI num_labelsrrYrZr[rrP classifierrcr\s r%rMz$LukeForEntityClassification.__init__2si  f%  ++zz&"<"<= ))F$6$68I8IJ r$rlrrmrkrrsrtrurrnrrr)r*rc||n|jj}|j||||||||| | | | d }|jdddddf}|j |}|j |}d}| | j |j} | jdk(r!tjj|| }nMtjj|jd| jdj|}|s5td|||j |j"|j$fDSt'|||j |j"|j$S) u entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)` or `(batch_size, num_labels)`, *optional*): Labels for computing the classification loss. If the shape is `(batch_size,)`, the cross entropy loss is used for the single-label classification. In this case, labels should contain the indices that should be in `[0, ..., config.num_labels - 1]`. If the shape is `(batch_size, num_labels)`, the binary cross entropy loss is used for the multi-label classification. In this case, labels should only contain `[0, 1]`, where 0 and 1 indicate false and true, respectively. Examples: ```python >>> from transformers import AutoTokenizer, LukeForEntityClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-open-entity") >>> model = LukeForEntityClassification.from_pretrained("studio-ousia/luke-large-finetuned-open-entity") >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [(0, 7)] # character-based entity span corresponding to "Beyoncé" >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: person ```NTrrrrac3$K|]}|| ywrr#rs r%r z6LukeForEntityClassification.forward..=r"r+r.r0rr1)r^rxrIrr[rrfrdndimrr cross_entropy binary_cross_entropy_with_logitsrrr(r0rr1r3r]rlrrmrkrrsrtrurrnrrr)r*rfeature_vectorr.r+s r%rqz#LukeForEntityClassification.forward>sm|&1%1559.2,0/3&*k E,,-k !!2!23k !!1!12 k u//0 k U--. k  ((9(9:k  ((8(89k &e&6&67k E--.k   1 12k **+k $D>k 'tnk d^k u00 1!k k r$rz The LUKE model with a classification head on top (a linear layer on top of the hidden states of the two entity tokens) for entity pair classification tasks, such as TACRED. c"eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e e ffdZ xZS)LukeForEntityPairClassificationc4t||t||_|j|_t j |j|_t j|jdz|jd|_ |jy)NrFrr\s r%rMz(LukeForEntityPairClassification.__init__sp  f%  ++zz&"<"<= ))F$6$6$:F>> from transformers import AutoTokenizer, LukeForEntityPairClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred") >>> model = LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred") >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [ ... (0, 7), ... (17, 28), ... ] # character-based entity spans corresponding to "Beyoncé" and "Los Angeles" >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: per:cities_of_residence ```NTrrrrrac3$K|]}|| ywrr#rs r%r z:LukeForEntityPairClassification.forward..%rr"r)r^rxrIr rrr[rrfrdrrrrrrrr(r0rr1r7rs r%rqz'LukeForEntityPairClassification.forwardsB&1%1559-1,0/3&*p E,,-p !!2!23p !!1!12 p u//0 p U--. p  ((9(9:p  ((8(89p &e&6&67p E--.p   1 12p ))*p $D>p 'tnp d^p u44 5!p p r$rz The LUKE model with a span classification head on top (a linear layer on top of the hidden states output) for tasks such as named entity recognition. c&eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd eejd eejd eejdeejdee dee dee de e e ff"dZ xZS)LukeForEntitySpanClassificationc2t||t||_|j|_t j |j|_t j|jdz|j|_ |jy)Nr rr\s r%rMz(LukeForEntitySpanClassification.__init__;sn  f%  ++zz&"<"<= ))F$6$6$:F>> from transformers import AutoTokenizer, LukeForEntitySpanClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-conll-2003") >>> model = LukeForEntitySpanClassification.from_pretrained("studio-ousia/luke-large-finetuned-conll-2003") >>> text = "Beyoncé lives in Los Angeles" # List all possible entity spans in the text >>> word_start_positions = [0, 8, 14, 17, 21] # character-based start positions of word tokens >>> word_end_positions = [7, 13, 16, 20, 28] # character-based end positions of word tokens >>> entity_spans = [] >>> for i, start_pos in enumerate(word_start_positions): ... for end_pos in word_end_positions[i:]: ... entity_spans.append((start_pos, end_pos)) >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_indices = logits.argmax(-1).squeeze().tolist() >>> for span, predicted_class_idx in zip(entity_spans, predicted_class_indices): ... if predicted_class_idx != 0: ... print(text[span[0] : span[1]], model.config.id2label[predicted_class_idx]) Beyoncé PER Los Angeles LOC ```NTrrarrrc3$K|]}|| ywrr#rs r%r z:LukeForEntitySpanClassification.forward..rr"r)r^rxrIr#rhrtrurdrfr gatherrrr[rrrrrrrrrr(r0rr1r9)r]rlrrmrkrrsrtrurrrrnrrr)r*rrP start_states end_statesrr.r+s r%rqz'LukeForEntitySpanClassification.forwardGs^&1%%>w?X?X?_?_%` "||G$=$=rCYZ 3==bAHHRQ\]  & &'*C*C*J*J J#7#:#:7;T;T;[;[#\ \\'";";RAUV L*g>^>^#_efgn50  YYv}}-F{{a}}226;;r4??3SU[U`U`acUde}}EEfkkRToW]WbWbceWfWnWnouWvw(=(=w?[?[]d]o]op  .!//!(!=!=))   r$)NNNNNNNNNNNNNNNN)rrrrMrrr rr!rrrr9rqrwrxs@r%rr4s 156:593715<@<@:>=A;?1559-1,0/3&*#H E,,-H !!2!23H !!1!12 H u//0 H U--. H  ((8(89H  ((8(89H &e&6&67H !))9)9 :H 'u'7'78H E--.H   1 12H ))*H $D>H 'tn!H "d^#H $ u44 5%H H r$rz The LUKE Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c"eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e e ffdZ xZS)LukeForSequenceClassificationc\t|||j|_t||_t j |j |jn |j|_ t j|j|j|_ |jyrrLrMrr^rIrrYclassifier_dropoutrZr[rrPrrcr\s r%rMz&LukeForSequenceClassification.__init__s   ++f% zz)/)B)B)NF % %TZTnTn  ))F$6$68I8IJ r$rlrrmrkrrsrtrurrnrrr)r*rc||n|jj}|j||||||||| | | | d }|j}|j |}|j |}d}| | j |j} |jj|jdk(rd|j_nl|jdkDrL| jtjk(s| jtjk(rd|j_nd|j_|jjdk(rIt}|jdk(r&||j| j}n||| }n|jjdk(r=t!}||j#d|j| j#d}n,|jjdk(rt%}||| }|s5t'd |||j(|j*|j,fDSt/|||j(|j*|j, S) a entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrr regressionsingle_label_classificationmulti_label_classificationrac3$K|]}|| ywrr#rs r%r z8LukeForSequenceClassification.forward..Brr"r)r^rxrIrwr[rrfrd problem_typerrcr rjrr squeezer rrr(r0rr1r;)r]rlrrmrkrrsrtrurrnrrr)r*rr:r.r+loss_fcts r%rqz%LukeForSequenceClassification.forwards#V&1%1559.2,0/3&*g E,,-g !!2!23g !!1!12 g u//0 g U--. g  ((9(9:g  ((8(89g &e&6&67g E--.g   1 12g **+g $D>g 'tng d^g u22 3!g g r$rz The LUKE Model with a token classification head on top (a linear layer on top of the hidden-states output). To solve Named-Entity Recognition (NER) task using LUKE, `LukeForEntitySpanClassification` is more suitable than this class. c"eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e e ffdZ xZS)LukeForTokenClassificationc`t|||j|_t|d|_t j |j |jn |j|_ t j|j|j|_ |jyNF)r_rr\s r%rMz#LukeForTokenClassification.__init__Ys   ++f> zz)/)B)B)NF % %TZTnTn  ))F$6$68I8IJ r$rlrrmrkrrsrtrurrnrrr)r*rcN||n|jj}|j||||||||| | | | d }|j}|j |}|j |}d}| W| j |j} t}||jd|j| jd}|s5td|||j|j|jfDSt|||j|j|jS)aM entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NTrrac3$K|]}|| ywrr#rs r%r z5LukeForTokenClassification.forward..rr"r)r^rxrIr#r[rrfrdr rrr(r0rr1r>)r]rlrrmrkrrsrtrurrnrrr)r*rrr.r+rs r%rqz"LukeForTokenClassification.forwardfs1V&1%rqrwrxs@r%rrQs} 156:593715=A<@:>1559.2,0/3&*U E,,-U !!2!23U !!1!12 U u//0 U U--. U  ((9(9:U  ((8(89U &e&6&67U E--.U   1 12U **+U $D>U 'tnU d^U u// 0!U U r$rc$eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd eejd eejd eejdee dee dee de e e ff dZ xZS)LukeForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyr) rLrMrr^rIrrrP qa_outputsrcr\s r%rMz!LukeForQuestionAnswering.__init__sU   ++f> ))F$6$68I8IJ r$rlrrmrkrrsrtrurrnstart_positions end_positionsrr)r*rc`||n|jj}|j||||||||| | | |d }|j}|j |}|j dd\}}|j d}|j d}d}| | t| jdkDr| j d} t| jdkDr| j d} |jd}| jd|| jd|t|}||| }||| }||zd z }|s6td ||||j|j|jfDSt||||j|j|j S) a  entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. NTrrrarr) ignore_indexrc3$K|]}|| ywrr#rs r%r z3LukeForQuestionAnswering.forward..s = r")r+rArBr0rr1)r^rxrIr#rsplitrlenrhclamp_r r(r0rr1r@)r]rlrrmrkrrsrtrurrnrrrr)r*rrr.rArB total_loss ignored_indexr start_lossend_losss r%rqz LukeForQuestionAnswering.forwardsP&1%15596:48,0/3&*!f E,,-f !!2!23f !!1!12 f u001 f U--. f  ((9(9:f  ((8(89f &e&6&67f E--.f   1 12f "%"2"23f   0 01f $D>f 'tnf d^!f " u66 7#f f r$rc"eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e e ffdZ xZS)LukeForMultipleChoicec&t||t||_t j |j |j n |j|_t j|jd|_ |jyr) rLrMr^rIrrYrrZr[rrPrrcr\s r%rMzLukeForMultipleChoice.__init__8so  f% zz)/)B)B)NF % %TZTnTn  ))F$6$6: r$rlrrmrkrrsrtrurrnrrr)r*rc:||n|jj}||jdn| jd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}| 1| jd| j d| j dnd} |!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|1|jd|j d|j dnd}|j ||||||||| | | | d }|j }|j|}|j|}|jd|}d}| .| j|j} t}||| }|s5td|||j|j|jfDSt!|||j|j|jS) a^ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NrrarTrc3$K|]}|| ywrr#rs r%r z0LukeForMultipleChoice.forward..r!r"r)r^rxrrrhrIrwr[rrfrdr r(r0rr1rD)r]rlrrmrkrrsrtrurrnrrr)r* num_choicesrr:r.reshaped_logitsr+rs r%rqzLukeForMultipleChoice.forwardDsF&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  BLAWZ__R)<=]a %0 " & &r+@+E+Eb+I J %0 " & &r+@+E+Eb+I J #. $ $R)<)A)A")EGZG_G_`bGc d  ))))%!"7"7 3'/!5   --  ]3 / ++b+6  YY556F')HOV4D #))00&&    -"!//!(!=!=))   r$r)rrrrMrrr rr!rrrrDrqrwrxs@r%rr6s} 156:593715=A<@:>1559.2,0/3&*P E,,-P !!2!23P !!1!12 P u//0 P U--. P  ((9(9:P  ((8(89P &e&6&67P E--.P   1 12P **+P $D>P 'tnP d^P u33 4!P P r$r) rrrrrrrrr^rH)Frr dataclassesrtypingrrrr torch.utils.checkpointrtorch.nnrr r activationsr r modeling_outputsrrmodeling_utilsr pytorch_utilsrutilsrrrconfiguration_luker get_loggerrloggerrr(r*r3r7r9r;r>r@rDr[rFrzrrrrrrrr4r<rArHr^rerrrrrrrrr__all__r#r$r%rs !)) AA'K-699*   H % I%?I I< I/I I< '?'? '?T ?? ?< ?[? ?< ?[? ?< ?;? ?B ? ? ?B !?{!? !?H  ?K ?  ?FF=RYYF=R(299(Vn nd RYY ,BII,` ryy   1 1hK "))K ^  BII" 299 */**0 Y'#Y'  Y'x 4"**> L )L  L ^ y "5y  y x ~ &9~  ~ B V &9V  V r u $7u  u pd !4d d Ns 2s s l^ /^ ^ B r$