UhQdZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZdd lmZmZmZmZdd lmZmZdd lmZmZdd lm Z m!Z!m"Z"m#Z#d dl$m%Z%e"jLe'Z(Gdde jRZ*Gdde jRZ+ d "55<!>z6SX!Y "44 \\*- r8)FNF)__name__ __module__ __qualname____doc__rboolr%r'TensorintrTr BoolTensorrd __classcell__r6s@r7rr*s ,z ,4 ,D ,&M5<<&M&MUX&M]b]i]i&MV7;). ll"%"2"23#'   r8rcZeZdZdZfdZdej dej fdZxZS)r-z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr) kernel_sizestride)r$r% image_sizer4 num_channelsr) isinstance collectionsabcIterabler/r Conv2d projection)r5rrtr4rur)r/r6s r7r%zDeiTPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir8rUr!c|j\}}}}||jk7r td|j|j dj dd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r#r)rEru ValueErrorr{flatten transpose)r5rUr]rur:r;xs r7rdzDeiTPatchEmbeddings.forwardsa2>2D2D/ L&% 4,, ,w  OOL ) 1 1! 4 > >q! Dr8) rfrgrhrir%r'rkrdrnros@r7r-r-s) jELLU\\r8r-modulequerykeyvalueattention_maskscalingr3c tj||jdd|z}tjj |dtj j|j}tjj|||j}|||z}tj||} | jddj} | |fS)Nr=)rDdtype)ptrainingrr#) r'matmulrr rJsoftmaxfloat32torr3r contiguous) rrrrrrr3kwargs attn_weights attn_outputs r7eager_attention_forwardrs<<s}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#n4 ,,|U3K''1-88:K  $$r8c eZdZdeddffd Zdej dej fdZ d deej de de e ej ej fe ej ffd Z xZ S) DeiTSelfAttentionrr!Nc2t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |j|_ |jdz|_ d|_ tj|j|j|j |_tj|j|j|j |_tj|j|j|j |_y) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)r$r%r)num_attention_headshasattrr}rrlattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalr Linearqkv_biasrrrr5rr6s r7r%zDeiTSelfAttention.__init__sF     : : :a ?PVXhHi"6#5#5"67334A7   #)#=#= #&v'9'9FO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=/2O\M]r8re)rfrgrhrr%r'rkrrrjrrrdrnros@r7rrs]z]d](%ell%u||% bg!(0(>!Z^! uU\\5<</0% 2EE F!r8rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) DeiTSelfOutputz The residual connection is defined in DeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rr!Nct|tj|j|j|_tj |j|_yN) r$r%r rr)denser1r2r3rs r7r%zDeiTSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r8r input_tensorcJ|j|}|j|}|Srrr3r5rrs r7rdzDeiTSelfOutput.forward s$ =1  ]3 r8) rfrgrhrirr%r'rkrdrnros@r7rrsD >z>d> U\\RWR^R^r8rc eZdZdeddffd ZdeeddfdZ d dejde ejd e de e ejejfe ejffd ZxZS) DeiTAttentionrr!Nct|t||_t ||_t |_yr)r$r%r attentionroutputset pruned_headsrs r7r%zDeiTAttention.__init__s0 *62$V, Er8headsc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)NrrrC)lenrrrrrrrrrrrrunion)r5rindexs r7 prune_headszDeiTAttention.prune_headss u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r8rrrch|j|||}|j|d|}|f|ddz}|S)Nrr)rr)r5rrr self_outputsattention_outputrs r7rdzDeiTAttention.forward.sE ~~mY@QR ;;|A F#% QR(88r8re)rfrgrhrr%rrlrr'rkrrjrrrdrnros@r7rrs"z"d" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F r8rc`eZdZdeddffd Zdej dej fdZxZS)DeiTIntermediaterr!Nct|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r$r%r rr)intermediate_sizerrv hidden_actstrrintermediate_act_fnrs r7r%zDeiTIntermediate.__init__>s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r8rcJ|j|}|j|}|Sr)rr)r5rs r7rdzDeiTIntermediate.forwardFs& =1 00? r8 rfrgrhrr%r'rkrdrnros@r7rr=s19z9d9U\\ellr8rcxeZdZdeddffd Zdej dej dej fdZxZS) DeiTOutputrr!Nct|tj|j|j |_tj|j|_ yr) r$r%r rrr)rr1r2r3rs r7r%zDeiTOutput.__init__OsB YYv779K9KL zz&"<"<= r8rrcT|j|}|j|}||z}|Srrrs r7rdzDeiTOutput.forwardTs. =1  ]3 % 4 r8rros@r7rrNs?>z>d> U\\RWR^R^r8rc eZdZdZdeddffd Z d dejdeejde de e ejejfe ejffd Z xZ S) DeiTLayerz?This corresponds to the Block class in the timm implementation.rr!Ncrt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nreps)r$r%chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr)layer_norm_epslayernorm_beforelayernorm_afterrs r7r%zDeiTLayer.__init__as '-'E'E$&v.,V4 ( " V-?-?VEZEZ [!||F,>,>FDYDYZr8rrrc|j|j|||}|d}|dd}||z}|j|}|j|}|j ||}|f|z}|S)N)rrr)rrrrr)r5rrrself_attention_outputsrr layer_outputs r7rdzDeiTLayer.forwardks "&  ! !- 0 /"0"  2!4(,)=8 ++M: ((6 {{<? /G+r8re)rfrgrhrirr%r'rkrrjrrrdrnros@r7rr^sI[z[d[-1"' ||ELL)  uU\\5<</0% 2EE F r8rceZdZdeddffd Z d dej deej deded ede e e ff d Z xZ S) DeiTEncoderrr!Nct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wre) r$r%rr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r5rr\r6s r7r%zDeiTEncoder.__init__sN  ]]uVE]E]?^#_!If$5#_` &+#$`sA#rrroutput_hidden_states return_dictct|rdnd}|rdnd}t|jD]h\}} |r||fz}|||nd} |jr+|jr|j | j || |} n | || |} | d}|s`|| dfz}j|r||fz}|st d|||fDSt|||S)Nrrc3&K|] }|| ywrr).0vs r7 z&DeiTEncoder.forward..smq_`_lms)last_hidden_stater attentions) enumeraterrr_gradient_checkpointing_func__call__tupler) r5rrrrrall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r7rdzDeiTEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO**t}} $ A A ))!#% ! !-]OM^ _ )!,M &9]1=MS>ST,=j(4  r8c.|jjSr)r9r.)r5s r7get_input_embeddingszDeiTModel.get_input_embeddingss///r8c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr$rrr)r5heads_to_prunerrs r7 _prune_headszDeiTModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr8rUrVrrrrrTc||n|jj}||n|jj}||n|jj}| t d|j ||jj }|jjjjj}|j|k7r|j|}|j|||} |j| ||||} | d} |j| } |j|j| nd} |s| | | fn| f} | | ddzSt!| | | j"| j$S)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rVrT)rrrrrr)r pooler_outputrr)rrruse_return_dictr} get_head_maskrr9r.r{rrrr$r%r'rrr)r5rUrVrrrrrTexpected_dtypeembedding_outputencoder_outputssequence_output pooled_output head_outputss r7rdzDeiTModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%:L?? /Tl+ ,, /!5# ' *!,..98< 8O O4UY ?L?XO];_n^pL/!""55 5)-')77&11   r8)TFNNNNNNF)rfrgrhrrjr%r-r*r.rrr'rkrmrrrrdrnros@r7r!r!szd[_lp&0&90C046:,0,0/3&*).; u||,; "%"2"23; ELL) ; $D> ; 'tn ; d^; #';  u00 1; ; r8r!c*eZdZdeffd ZdZxZS)r&rct|tj|j|j |_t|j|_ yr) r$r%r rr)pooler_output_sizerr pooler_act activationrs r7r%zDeiTPooler.__init__<s> YYv1163L3LM  !2!23r8c\|dddf}|j|}|j|}|S)Nr)rr>)r5rfirst_token_tensorr7s r7rdzDeiTPooler.forwardAs6+1a40 #56  6 r8)rfrgrhrr%rdrnros@r7r&r&;s4z4 r8r&a\ DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://arxiv.org/abs/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introceZdZdeddffd Ze d deejdeejdeejdee d ee d ee d e de e e ffd ZxZS)DeiTForMaskedImageModelingrr!NcNt||t|dd|_t j t j |j|jdz|jzdt j|j|_ |jy)NFT)r"r r#r) in_channels out_channelsrr) r$r%r!r r Sequentialrzr)encoder_strideru PixelShuffledecoderr(rs r7r%z#DeiTForMaskedImageModeling.__init__Ws  fdS }} II"..#22A58K8KK  OOF11 2    r8rUrVrrrrrTc ||n|jj}|j|||||||}|d} | ddddf} | j\} } } t | dzx} }| j dddj | | | |} |j| }d}||jj|jjz}|j d||}|j|jjdj|jjdjdj}tjj||d }||zj!|j!d zz |jj"z }|s|f|ddz}||f|zS|St%|||j&|j( S) a; bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```N)rVrrrrrTrrr=r>r#none) reductiongh㈵>)lossreconstructionrr)rr1r rErlrIrHrJrtr4repeat_interleaverZrr rJl1_losssumrurrr)r5rUrVrrrrrTrr6r]sequence_lengthrur:r;reconstructed_pixel_valuesmasked_im_lossr@r`reconstruction_lossrs r7rdz"DeiTForMaskedImageModeling.forwardhsL&1% R 'tn R d^R #'R  u// 0R R r8rCz DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. ceZdZdeddffd Ze d deejdeejdeejdee d ee d ee d e de e e ffd Z xZS)DeiTForImageClassificationrr!Nc.t|||j|_t|d|_|jdkDr*t j |j|jnt j|_ |jyNF)r"r) r$r% num_labelsr!r r rr)Identity classifierr(rs r7r%z#DeiTForImageClassification.__init__ss   ++f> OUN_N_bcNc"))F$6$68I8IJikititiv r8rUrlabelsrrrrTcb||n|jj}|j||||||}|d} |j| dddddf} d} ||j | j }|jj |jdk(rd|j_nl|jdkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt} |jdk(r&| | j|j} n| | |} n|jj dk(r=t} | | jd|j|jd} n,|jj dk(rt!} | | |} |s| f|ddz} | | f| zS| St#| | |j$|j& S) aZ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a DeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: Polaroid camera, Polaroid Land camera ```NrrrrrTrr regressionsingle_label_classificationmulti_label_classificationr=)rNlogitsrr)rr1r r]rdevice problem_typer[rr'longrlr squeezer rLr rrr)r5rUrr^rrrrTrr6rdrNloss_fctrs r7rdz"DeiTForImageClassification.forwardsT&1% Y 'tn Y d^Y #'Y  u++ ,Y Y r8rXceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<y)+DeiTForImageClassificationWithTeacherOutputa5 Output type of [`DeiTForImageClassificationWithTeacher`]. Args: logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nrd cls_logitsdistillation_logitsrr)rfrgrhrirdrr' FloatTensor__annotations__rlrmrrrrr8r7rkrk.s},+/FHU&& '..2J**+27;%"3"34;8