UhddlmZmZmZmZddlZddlmZddlmZddl m Z m Z m Z m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,e)r ddl-m.Z.ddl/m0Z0e*jbe2Z3GddejhZ5dZ6dejnde8dejnfdZ9 d=dejhdejndejndejnd eejnd!e:d"e:fd#Z;d>d$Z<Gd%d&ejhZ=ed'Gd(d)ejhZ>Gd*d+eZ?e'Gd,d-e"Z@Gd.d/ejhZAe'Gd0d1e@ZBGd2d3ee&ZCe'Gd4d5e@eZDe'd67Gd8d9e@ZEe'Gd:d;e@ZFgd<ZGy)?)CallableOptionalTupleUnionN)nn)ACT2FN)Cache DynamicCacheSlidingWindowCache StaticCache)GenerationMixin)use_kernel_forward_from_hub)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPastTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging) Phi3Config) BlockMask)make_flex_block_causal_maskcVeZdZfdZdej dej fdZxZS)Phi3MLPc*t|||_tj|j d|j zd|_tj|j |j d|_t|j|_ y)NFbias) super__init__configrLinear hidden_sizeintermediate_size gate_up_proj down_projr hidden_act activation_fnselfr- __class__s x/var/www/catia.catastroantioquia-mas.com/valormas/lib/python3.12/site-packages/transformers/models/phi3/modeling_phi3.pyr,zPhi3MLP.__init__:sp  IIf&8&8!f>V>V:V]bc6#;#;V=O=OV[\#F$5$56 hidden_statesreturnc|j|}|jdd\}}||j|z}|j|S)Nr(dim)r1chunkr4r2)r6r: up_statesgates r8forwardzPhi3MLP.forwardBsL%%m4 #//!/4i 2 24 88 ~~i((r9)__name__ __module__ __qualname__r,torch FloatTensorrC __classcell__r7s@r8r&r&9s'7)U%6%6)5;L;L)r9r&c|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr=r(r>)shaperGcat)xx1x2s r8 rotate_halfrQKsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r9r:n_repr;c|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)rLexpandreshape)r:rRbatchnum_key_value_headsslenhead_dims r8 repeat_kvrZRso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr9modulequerykeyvalueattention_maskscalingdropoutc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr(rr=)r?dtype)ptrainingr!)rZnum_key_value_groupsrGmatmul transposerLr functionalsoftmaxfloat32tordrarf contiguous) r[r\r]r^r_r`rakwargs key_states value_states attn_weights causal_mask attn_outputs r8eager_attention_forwardru^s3 ; ;) unsqueezerLrGrMrQ) qkcossin position_ids unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds r8apply_rotary_pos_embrxs( -- &C -- &C2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Eii%#++e*eZdZdZd dedeeffd Z ddejde ejejfdeejdee d eejd e ed e ejeejee ejffd ZxZS) Phi3Attentionz=Multi-headed attention from 'Attention Is All You Need' paperr- layer_idxc|t|||_||_t |d|j |j z|_|j |jz|_ |j|_|jdz|_ |j|_ d|_ |j |jzd|j|jzzz}tj|j |jz|j d|_tj|j |d|_y)NrYgTr(Fr))r+r,r-rgetattrr/num_attention_headsrYrWrgr`attention_dropout is_causalrr.o_projqkv_proj)r6r-rop_sizer7s r8r,zPhi3Attention.__init__s  " F4F4F&JdJd4de $*$>$>&B\B\$\!#)#=#= }}d* !'!9!9,,t}}d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL"4;;0@$G %  % ! \*k));;;;FFH kk+. L((r9N)NN)rDrErF__doc__r"rintr,rGTensorrr LongTensorrrrCrIrJs@r8rrsG Kz Khsm K(+/59 6)||6)#5<<#=>6)!. 6) ! 6) !!1!12 6)-.6) u||Xell3XeELL>Q5RR S6)r9rRMSNormc,eZdZdfd ZdZdZxZS) Phi3RMSNormct|tjt j ||_||_y)z: Phi3RMSNorm is equivalent to T5LayerNorm N)r+r,r ParameterrGonesweightvariance_epsilon)r6r/epsr7s r8r,zPhi3RMSNorm.__init__s1 ll5::k#:; #r9c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)Nr(r=T)keepdim) rdrmrGrlpowmeanrsqrtrr)r6r: input_dtypevariances r8rCzPhi3RMSNorm.forwardsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r9c^t|jjd|jS)Nz, eps=)tuplerrLrr6s r8 extra_reprzPhi3RMSNorm.extra_reprs* ))*+6$2G2G1HIIr9)gư>)rDrErFr,rCrrIrJs@r8rrs$;Jr9rcpeZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee ejejfd e ed e ejee ejejfffdZxZS)Phi3DecoderLayerr-rct||j|_t|||_t ||_t|j|j|_ t|j|j|_ ||_ tj|j|_tj|j|_y)N)r-rr)r+r,r/r self_attnr&mlpr rms_norm_epsinput_layernormpost_attention_layernormr-rDropout resid_pdropresid_attn_dropoutresid_mlp_dropoutr6r-rr7s r8r,zPhi3DecoderLayer.__init__s !--&f J6?*6+=+=6CVCVW(3F4F4FFL_L_(`% "$**V-?-?"@!#F,>,>!?r9r:r_r|rr use_cacherrror;c |} |j|}|jd||||||||d| \}} | |j|z}|} |j|}|j |}| |j |z}|f} |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_value (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r:r_r|rrrrr)rrrrrr) r6r:r_r|rrrrrroresidualself_attn_weightsoutputss r8rCzPhi3DecoderLayer.forwardsD!,,]; ,:4>> , ')%)/) 3 ,  , ( (!4#:#:=#II  55mD /  4#9#9-#HH  "  )+ +Gr9)NNNFFNN)rDrErFr"rr,rGrrrr boolrrrrHrCrIrJs@r8rrs @z @c @2637*.,1$)59KO=||=!.=u//0 = ! = $D> =D>=!!1!12=&eELL%,,,F&GH=-.= u  (51B1BEDUDU1U+V"WW X=r9rcJeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZdZdZdZy)Phi3PreTrainedModelmodelTrpast_key_valuesz0.0.5c|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyt|tr&|j j jdyy)Nr)rstdg?)r-initializer_range isinstancerr.rdatanormal_r*zero_ Embedding padding_idxrfill_)r6r[rs r8 _init_weightsz!Phi3PreTrainedModel._init_weightsTskk++ fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  , MM   $ $S )-r9N)rDrErFr" config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_attention_backend_versionrrr9r8rrDsXL&*#+,#4"5!N  $!"&H *r9rc^eZdZddeffd Zej edZxZ S)Phi3RotaryEmbeddingr-ct|t|drG|j;|jj d|jj d|_nd|_|j |_|j |_||_ t|j |_ |j|j|\}|_ |jd|d|j|_y)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r+r,hasattrrrrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr-r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r6r-devicerr7s r8r,zPhi3RotaryEmbedding.__init__cs  6> *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+)rd)rfloatrTrLrmrrrstrrGautocastrirMrzrr{rd) r6rNr|inv_freq_expandedposition_ids_expandedrfreqsembrzr{s r8rCzPhi3RotaryEmbedding.forwardtsV!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.r) rDrErFr"r,rGno_gradrrCrIrJs@r8rrbs3/z/"U]]_ < ?ab b   --i8M 0*nO  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L.. M>?L] & #oom\J#7BD0d![[)H4;;+H+HI 6M#!m%55!) *)."3#-$7 $ M*!,M =#3"55' 6* -0  -!1 1 &+/8Od+%   r9r# input_tensorc l|jjdk(rS|H|F|dddfjj|j dk7}|r t d|d|vr|Sy|jjdk(r't |tjr t|}|S||jnd}t |t}t |t} |jjdk(r?|s=| s;|s9tj||||jj|j ry|j"} tj$| j&} |j(d } | s|r|j+} n1t |tjr|j(dn|| zd z} |j-|| | | ||j(d|j| }|jjdk(r2|0|j.j0d vr|stj2|| }|S) Nflash_attention_2r=rzYou are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Phi3. Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input. rflex_attentionr)r%past_key_values_lengthr is_trainingr!)sequence_length target_lengthrdr batch_sizer-r)cudaxpunpu)r-rsumitemsizer,rrGrr$r-r r r_ignore_causal_mask_sdparrfrdfinfominrLget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrr_unmask_unattended)r6r_r5rrris_padding_rightr0using_static_cacheusing_sliding_window_cacherd min_dtyper;r<rss r8r/zPhi3Model._update_causal_masks: ;; + +/B B)o.I#1!R%#8#<#<#>#C#C#EIZIZI\]^I_#_ #$a )c^.C%% ;; + +/? ?.%,,7!<^!L! ! @O?Z?99;`a'E%/AS%T" KK , , 6'+E%%>>*'7#{{99 MM ""KK&** &,,Q/ %);+??AM nell;$$R(%7!; PP +')#))!,;;+Q   KK , , 6*%%**.DD% 1CCKQZ[Kr9r;r<rdr=c||jdk(r|}|Stj|j} tj||f| ||j }tj ||j |jddkD} |j} t| ddrs| jgt|tr||kDrRtj ||j |jdd| jz k} | j| || z}|ddddddfj|ddd}||j}|j d|kDr |ddd|f}|j d} |ddddddd| f|ddddddfj#|j z}|d k(}|ddddddd| fj%|| |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. config (`Phi3Config`): The model's configuration class past_key_values (`Cache`): The cache class that is being used currently to generate N) fill_valuerdrr)r=r!use_sliding_windowTr)r?rGrErFfullrr.rUget_text_configrrrr bitwise_or_rTclonerLrm masked_fill)r_r;r<rdrr=r-rrsrMdiagonal_attend_mask text_configsliding_attend_mask mask_length padding_masks r8rHz?Phi3Model._prepare_4d_causal_attention_mask_with_cache_positionRs%B  %.*<*<*>!*C(K@= E*..I** -0Ye\j\q\qK$)<< nF[F[#\_m_u_uA`$ !002K{$8$?KD^D^Dj"/3EF/\iJi*/,,}^MbMb*c&..r15 8R8RR+')445HI / /K%dD!Q&67>>z1bRTUK))//1 !''+m;%3A~ ~4E%FN,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r9 NNNNNNNNN)F)rDrErFr"r,r r#rrrrGrrr rHrrrrrCrr/ staticmethodrrdrHrIrJs@r8rrsz !"151537+/59$(,0/359\ E,,-\ !.\ u//0 \ "% \   1 12 \ D>\ $D>\ 'tn\ !!1!12\ $$89\  !\ \ H#( TellK78TllT T  T  TlB BBB{{ B  B  BBBBr9rc eZdZy)KwargsForCausalLMN)rDrErFrr9r8r_r_sr9r_ceZdZdgZddiZddgdgfiZfdZdZdZd Z d Z d Z d Z e e dd eej"deej$deej"deedeej(deej"deedeedeedeej"deeej$fdeedefdZ dfd ZxZS)Phi3ForCausalLMzlm_head.weightlm_head colwise_repr:logitsct||t||_|j|_t j |j|jd|_|jyNFr)) r+r,rrrrr.r/rbrr5s r8r,zPhi3ForCausalLM.__init__sU  v&  ++yy!3!3V5F5FUS  r9c.|jjSrrrrs r8r z$Phi3ForCausalLM.get_input_embeddingszz&&&r9c&||j_yrrhr"s r8r#z$Phi3ForCausalLM.set_input_embeddings"' r9c|jSrrbrs r8get_output_embeddingsz%Phi3ForCausalLM.get_output_embeddingss ||r9c||_yrrm)r6new_embeddingss r8set_output_embeddingsz%Phi3ForCausalLM.set_output_embeddingss % r9c||_yrr)r6decoders r8 set_decoderzPhi3ForCausalLM.set_decoders  r9c|jSrrsrs r8 get_decoderzPhi3ForCausalLM.get_decoders zzr9r$r_r|rr%labelsrrr&rlogits_to_keepror;c ||n|jj}| | n|jj} |jd|||||||| | d | } | j}t | t r t| dn| }|j|dd|ddf}d}|*|jd|||jjd| }t||| j| j| jS)an labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Phi3ForCausalLM >>> model = Phi3ForCausalLM.from_pretrained("meta-phi3/Phi3-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-phi3/Phi3-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) r$r_r|rr%rrr&r)rdrxrlossrdrr:r+r)r-rr&rr*rrslicerb loss_functionrrrr:r+)r6r$r_r|rr%rxrrr&rryrorr: slice_indicesrdr|s r8rCzPhi3ForCausalLM.forwardsN2C1N-TXT_T_TqTq$8$D $++JjJj  ,64:: , )%+'/!5) ,  ,  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD%#33!//))   r9c |r_|jjrI|jd|jjdzk\r |d} | |jjkrd}t |d||||||||d| } | S)Nr!r)r$rr_r%rr|rryr)r-rrL original_max_position_embeddingsr+prepare_inputs_for_generation) r6r$rr_r%rr|rryro past_length model_inputsr7s r8rz-Phi3ForCausalLM.prepare_inputs_for_generations$  (("dkk&R&RUV&VV(+KdkkJJJ"&w<  +)')%)    r9) NNNNNNNNNNr)NNNNNTN)rDrErF_tied_weights_keys_tp_plan_pp_planr,r r#rnrqrurwrrrrGrrr rHrrrrr_rrCrrIrJs@r8raras*+=)H_-z:;H'(&151537+/59-1$(,0/35934G E,,-G !.G u//0 G "% G   1 12 G ))*G D>G $D>G 'tnG !!1!12G c5<</0G *+G  G G X%%r9raa The Phi3 Model transformer with a sequence classification head on top (linear layer). [`Phi3ForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introceZdZfdZdZdZee ddee jdee jdee jdee dee jd ee jd eed eed eed efdZxZS)Phi3ForSequenceClassificationct|||j|_t||_t j |j|jd|_|jyrf) r+r, num_labelsrrrr.r/scorerr5s r8r,z&Phi3ForSequenceClassification.__init__>sS   ++v& YYv114??O  r9c.|jjSrrhrs r8r z2Phi3ForSequenceClassification.get_input_embeddingsGrir9c&||j_yrrhr"s r8r#z2Phi3ForSequenceClassification.set_input_embeddingsJrkr9r$r_r|rr%rxrrr&r;c |j|||||||| } | j} |j| } ||jd} n|jd} |jj | dk7r t d|jj d}n|||jj k7j| jtj}tj|jd| jtj}||zjd}n.d}tj|jj d| tj| | j |f}d}||j#| |||j }t%||| j&| j(| j* S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). r_r|rr%rrr&Nrr!z=Cannot handle batch sizes > 1 if no padding token is defined.r=)rrdz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r))rdrx pooled_logitsr-r{)rr*rrLr-rr,rmrrGint32r.argmaxrrr7rDr~rrr:r+)r6r$r_r|rr%rxrrr&transformer_outputsr:rdr=last_non_pad_token non_pad_mask token_indicesrr|s r8rCz%Phi3ForSequenceClassification.forwardMs*8!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   %%VFR_hlhshs%tD/ /??-;;*55   r9r\)rDrErFr,r r#rrrrGrrr rHrrrCrIrJs@r8rr/s'(151537+/59-1$(,0/3A E,,-A !.A u//0 A "% A   1 12 A ))*A D>A $D>A 'tnA  *A A r9rceZdZfdZdZdZee ddee jdee jdee jdee dee jd ee jd eed eed eed efdZxZS)Phi3ForTokenClassificationct|||j|_t||_t |dd |j }nt |dd |j}nd}tj||_ tj|j|j|_ |jy)Nclassifier_dropouthidden_dropoutg?)r+r,rrrrrrrrrar.r/rr)r6r-rr7s r8r,z#Phi3ForTokenClassification.__init__s   ++v& 6/ 6 B!'!:!:  V-t 4 @!'!6!6 !$ zz"45 YYv1163D3DE  r9c.|jjSrrhrs r8r z/Phi3ForTokenClassification.get_input_embeddingsrir9c&||j_yrrhr"s r8r#z/Phi3ForTokenClassification.set_input_embeddingsrkr9r$r_r|rr%rxrrr&r;c |j|||||||| } | j} |j| } |j| } d} ||j | ||j } t | | | j| jS)rrN)r|rdr:r+) rr*rarr~r-rr:r+)r6r$r_r|rr%rxrrr&rsequence_outputrdr|s r8rCz"Phi3ForTokenClassification.forwards*,0:: )%+'/!5,6 , "33,,7O,  %%ffdkkBD$!//))   r9r\)rDrErFr,r r#rrrrGrrr rHrrrCrIrJs@r8rrs '(151537+/59-1$(,0/3* E,,-* !.* u//0 * "% *   1 12 * ))** D>* $D>* 'tn*  * * r9r)rrrarr)r)Nr!)HtypingrrrrrGr activationsr cache_utilsr r r r generationr integrationsrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrr configuration_phi3r"!torch.nn.attention.flex_attentionr#integrations.flex_attentionr$ get_loggerrDrModuler&rQrrrZrrurrrrrrrr_rarr__all__rr9r8rs(.43 !OO)7>B9 LF&hh* !;J   H %)bii)$( UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%4@H)BIIH)VY'J"))J(J(I1IX*/**:<"))P)?PPf  S $7S  S lC !4C C L r9