iU~ddlZddlmZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZmZmZdd lmZmZdd lmZdd lmZmZmZmZdd lmZddlmZmZddl m!Z!m"Z"ejFe$Z% d*dejLdejNdejNdejNdejNdzde(dzde(fdZ)GddejLZ*Gdde Z+eGddeZ,ed !Gd"d#e,Z-Gd$d%ejLZ.ed&!Gd'd(e,e Z/gd)Z0y)+N)Callable)nn)ACT2FN)Cache)GenerationMixin)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)check_model_inputs) AutoModelAutoModelForCausalLM) VoxtralConfigVoxtralEncoderConfigmodulequerykeyvalueattention_maskscalingdropoutc ||jddz}tj||jdd|z}|0|jdk(r!||ddddddd|j dfz}t jj|d}t jj|||j}tj||} | jd dj} | |fS) Nrr)dimptrainingr) sizetorchmatmul transposendimshaper functionalsoftmaxr!r* contiguous) rrrrrr r!kwargs attn_weights attn_outputs v/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/voxtral/modeling_voxtral.pyeager_attention_forwardr8,s**R.D(<<s}}Q':;gEL!n&9&9Q&>#nQ1o " o5M&NN ==((2(>L==((6??([L,,|U3K''1-88:K  $$c,eZdZdZ ddedededededed edzd edzffd Zd e jd edefdZ dde jde jdzdede e je jdze e jdzffdZ xZS)VoxtralAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsr! is_decoderbias is_causal layer_idxconfigc zt |||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ |/|r-tjd|jjd||_t!j"||d|_t!j"||||_t!j"||||_t!j"||||_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r$zInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Fr?)super__init__r<r=r!head_dimrB ValueErrorr r>r@logger warning_once __class____name__rArLineark_projv_projq_projout_proj) selfr<r=r!r>r?r@rArBrKs r7rFzVoxtralAttention.__init__Is  "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"     *4>>+B+B*CD,,  #ii 95A ii 94@ ii 94@  )YTB r9tensorseq_lenbszc|j|||j|jjddj S)Nrr)viewr=rGr.r3)rRrSrTrUs r7_shapezVoxtralAttention._shapeqs7{{3GQQRSUVWbbddr9 hidden_statesroutput_attentionsreturnc 0|j\}}}|j|j||jz||}|j|j |d|} |j|j |d|} t j|jjt} | ||| | |f|jsdn |jd|d|\} } | j||dj} |j| } | | fS)z#Input shape: Batch x Time x Channelr#?)r!r rZ)r+rXrPr rNrOr get_interfacerB_attn_implementationr8r*r!reshaper3rQ)rRrYrrZr4rUtgt_len_ query_states key_states value_statesattention_interfacer6r5s r7forwardzVoxtralAttention.forwardts(,,.Wa{{4;;}#= #LgWZ[ [[]!;RE {{4;;}#=r3G (?(M(M KK , ,.E) %8      % $}}C$,,/ %  % ! \"))#w;FFH mmK0 L((r9)r]FTFNNNF)rL __module__ __qualname____doc__intfloatboolrrFr,TensorrXtuplerh __classcell__rKs@r7r;r;Fs G   $'+&C&C&C &C  &C  &C&C:&C$&CPeU\\eCece /3"' ')||') t+') ') u||U\\D0% 2E2LL M ')r9r;c |eZdZdeffd Z ddej dej dedej fdZxZ S) VoxtralEncoderLayerrBcht||j|_t |j|j |j ||_tj|j|_ |j|_ t|j|_|j|_tj |j|j"|_tj |j"|j|_tj|j|_y)N)r<r=r!rB)rErFd_modelr<r;encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normr!ractivation_function activation_fnactivation_dropoutrMencoder_ffn_dimfc1fc2final_layer_normrRrBrKs r7rFzVoxtralEncoderLayer.__init__s )nn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r9rYrrZr[c|}|j|}|j|||\}}tjj ||j|j }||z}|}|j |}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|jtjk(rEtj|jjdz }tj || |}||fS)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rYrrZr(i)minmax)r|rzrr1r!r*rr~rrrdtyper,float16finforclamp)rRrYrrZresidualr5 clamp_values r7rhzVoxtralEncoderLayer.forwardsP!11-@ &*nn')/'5' # |  --mt||VZVcVc-d  =0  --m< **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0   %-- /++m&9&9:>>EK!KK K<[YMl**r9)F) rLrjrkrrFr,rprorhrrrss@r7rurusK=}=,#( %+||%+ %+ %+  %+r9rucDeZdZUeed<dZdZdZdZdZ dZ dZ dZ dZ dZdZy)VoxtralPreTrainedModelrBmodel)audiotextTNpast_key_values)rLrjrkr__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_cache_class_supports_attention_backend_can_compile_fullgraphr9r7rrsI (&*#"3N "&!r9rz: The Voxtral encoder, which is a Whisper encoder.  custom_introceZdZUdZeed<dZdZdgZe e dZ deffd Z dZ d ejfd Zd ejfd Ze dd eed eezfdZdej2fdZxZS)VoxtralEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`VoxtralEncoderLayer`]. Args: config: VoxtralEncoderConfig rBinput_featuresrru) attentionsrYcbt|||j|_|j|_|j }|j |_|j|_|jrtj|nd|_ tj|j |dd|_tj||ddd|_tj |j||_|j"j%dtj&t)|j*Dcgc] }t-|c}|_tj0|j |_tj4dd|_d|_|j;ycc}w) Nr^rr) kernel_sizepaddingr)rstriderF)r)rErFr!encoder_layerdrop layerdroprw num_mel_binsmax_source_positionsscale_embeddingmathsqrt embed_scalerConv1dconv1conv2 Embeddingembed_positionsrequires_grad_ ModuleListrangeencoder_layersrulayersr{ layer_norm AvgPool1d avg_poolergradient_checkpointing post_init)rRrBr<rcrKs r7rFzVoxtralEncoder.__init__s6  ~~ 11NN "//$*$?$?!393I3I499Y/sYYt00)TUV YYy)1VWX !||D,E,EyQ ++E2mm%PVPePeJf$gQ%8%@$gh ,,v~~6,,q3&+# %hs6F,cJ|jD] }d|_ d|_yri) parameters requires_grad_requires_grad)rRparams r7_freeze_parametersz!VoxtralEncoder._freeze_parameterss(__& (E"'E  (#r9r[c|jSNrrRs r7get_input_embeddingsz#VoxtralEncoder.get_input_embeddingss zzr9rc||_yrrrRrs r7set_input_embeddingsz#VoxtralEncoder.set_input_embeddings"s  r9r4c |jj|jjdz|jjdz}|j d|k7r"t d|d|j dd|d|j|jjj|jjj}tjj|j|}tjj|j |}|jddd }|jj}||zj|j}tjj!||j |j" }t%|j&D]\}} | || } | d}|j)|}t+| S) a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rr#z7Voxtral expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .)rdevicerrr()r)last_hidden_state)rBrrrrr0rHtoweightrrrr1gelupermuterr!r* enumeraterrr ) rRrrr4expected_seq_length inputs_embeds embed_posrYidx encoder_layer layer_outputss r7rhzVoxtralEncoder.forward%s&#kk>>ARARSTAUUX\XbXbXiXijkXll    #': :IJ]I^^jkykkAClDkEErsFrGGHI (**1B1B1H1HQUQ[Q[QbQbQiQi*j **4::n+EF  **4::m+DE %--aA6 ((// &266}7J7JK  --mt||VZVcVc-d "+DKK"8 - C)-M*!,M  - 6 )+  r9 input_lengthsc6|dz dzdz}|dz dzdz}||fS)zs Computes the output length of the convolutional layers and the output length of the audio encoder rrr)rRroutput_lengthss r7 _get_feat_extract_output_lengthsz/VoxtralEncoder._get_feat_extract_output_lengthsUs7'*q014 '!+1A5n,,r9r)rLrjrkrlrrmain_input_namerrr;ru_can_record_outputsrFrrModulerrrrrrqr rhr, LongTensorrrrrss@r7rrs  ! &O./&, 32$ bii")), +, , + + , , ^-e>N>N-r9rc*eZdZdeffd ZdZxZS)VoxtralMultiModalProjectorrBcft|tj|jj |j jd|_t|j|_ tj|j j|j jd|_ y)NFrD) rErFrrM audio_configintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rs r7rFz#VoxtralMultiModalProjector.__init___sz  &"5"5"G"GI[I[IgIgnst &556 &"4"4"@"@&BTBTB`B`glm r9cl|j|}|j|}|j|}|Sr)rrr)rRaudio_featuresrYs r7rhz"VoxtralMultiModalProjector.forwardes2 n5 /  m4 r9)rLrjrkrrFrhrrrss@r7rr^sn}n r9rzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a LLama language model. ceZdZdgZfdZdZdZdZdZdZ dZ e e d d e jd eed eezfdZe e dde j(dzd e jdzde j*dzde j(dzdedzde jdzde j(dzdedzde j(dzdee j*zd eed efdZfdZxZS)VoxtralForConditionalGenerationrc*t|||jj|_t j |j |_tj |j|_ t||_ |jyr) rErFr vocab_sizer from_configr audio_towerrlanguage_modelrmulti_modal_projectorrrs r7rFz(VoxtralForConditionalGeneration.__init__tsn   ,,77$001D1DE2>>v?Q?QR%?%G" r9c6|jjSr)rrrs r7rz4VoxtralForConditionalGeneration.get_input_embeddings~s""7799r9c:|jj|yr)rrrs r7rz4VoxtralForConditionalGeneration.set_input_embeddingss 007r9c6|jjSr)rget_output_embeddingsrs r7rz5VoxtralForConditionalGeneration.get_output_embeddingss""88::r9c:|jj|yr)rset_output_embeddings)rRnew_embeddingss r7rz5VoxtralForConditionalGeneration.set_output_embeddingss 11.Ar9c:|jj|yr)r set_decoder)rRdecoders r7rz+VoxtralForConditionalGeneration.set_decoders ''0r9c6|jjSr)r get_decoderrs r7rz+VoxtralForConditionalGeneration.get_decoders""..00r9zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.rrr4r[c |j|fddi|}|j}|jd|jjj }|j |}||_|S)aa input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] return_dictTr#)rrrarBrrr pooler_output)rRrr4 audio_outputsaudio_hidden_states audio_embedss r7get_audio_featuresz2VoxtralForConditionalGeneration.get_audio_featuressn)((TTTVT +==199"dkk>V>V>h>hi112EF &2 #r9N input_idsr position_idsrrlabels use_cachecache_positionlogits_to_keepc ||j|}|||j|dj} ||jjk(j d} |j | j|j| j|j}|jd||||||| | d| }|S)aj Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```T)r r#)rrrrrrrrr) rrr rBaudio_token_id unsqueezemasked_scatterrrr)rRrrrrrrrrrrr4r audio_token_maskoutputss r7rhz'VoxtralForConditionalGeneration.forwardsb  7D557 BM  %)*?22>t2TbbL!*T[[-G-G GRRSUV )88 ##M$8$89+> , )%+')) ,  , r9c|jdd}|jdd}t| |i|}|s|jdds||d<|S)Nris_first_iterationFrT)popgetrEprepare_inputs_for_generation)rRargsr4rr model_inputsrKs r7rz=VoxtralForConditionalGeneration.prepare_inputs_for_generations] $4d;#ZZ(r2sd, $ ! )9kkF&RR/2F   H %!% II% <<% % << % LL4' % T\ %%4U)ryyU)p8+48+v "_ " " m-+m-  m-`   L&