iL ZdZddlmZddlmZddlZddlmZmZddlm Z ddl m Z dd l mZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZmZmZmZm Z ddl!m"Z"m#Z#m$Z$erddl%m&Z&ejNe(Z)dej dej fdZ*dej dej fdZ+eeGddeZ,dedefdZ-dedefdZ.dZ/dZ0eed Gd!d"eZ1eed# Gd$d%eZ2Gd&d'ejfZ4Gd(d)ejfZ5Gd*d+ejfZ6Gd,d-ejfZ7Gd.d/eZ8eGd0d1eZ9Gd2d3ejfZ:Gd4d5ejfZ;Gd6d7e9Z<Gd8d9ejfZ=Gd:d;e9Z>eGd<d=e9Z?Gd>d?ejfZ@Gd@dAejfZAGdBdCe9ZBgdDZCy)EzPyTorch OWL-ViT model.) dataclass)AnyN)Tensornn)initialization)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleis_vision_availablelogging torch_int) OwlViTConfigOwlViTTextConfigOwlViTVisionConfig)center_to_corners_formatlogitsreturnctjj|tjt ||j S)Ndevice)r functional cross_entropytorcharangelenr!)rs t/home/obispo/Crisostomo_bridge/mision_env/lib/python3.12/site-packages/transformers/models/owlvit/modeling_owlvit.pycontrastive_lossr(4s/ == & &vu||CKPVP]P]/^ __ similaritycZt|}t|j}||zdz S)Ng@)r(t)r* caption_loss image_losss r' owlvit_lossr/9s,#J/L!*,,.1J : % ,,r)ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ e ed<dZe ed <d eefd Zy) OwlViTOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size * num_max_text_queries, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`OwlViTTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`OwlViTVisionModel`]. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrcHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw)r7r8Ngetattrto_tuple.0kselfs r' z(OwlViTOutput.to_tuple.._=  LLDGRYZ^`aRbRkRkRm m -0tuplekeysrBs`r'r>zOwlViTOutput.to_tuple^# YY[   r))__name__ __module__ __qualname____doc__r2r$ FloatTensor__annotations__r3r4r5r6r7rr8rGrr>r)r'r1r1?s(&*D%  d ")15e''$.504OU&&-4,0K""T)0-1L%##d*148186:3: %* r)r1r,c|jr>|jtjtjfvr|S|j S|jtj tjfvr|S|jSN) is_floating_pointdtyper$float32float64floatint32int64int)r,s r'_upcastr\fs`GG u}}==qL1779LGG U[[99qFquuwFr)boxescft|}|dddf|dddfz |dddf|dddfz zS)a Computes the area of a set of bounding boxes, which are specified by its (x1, y1, x2, y2) coordinates. Args: boxes (`torch.FloatTensor` of shape `(number_of_boxes, 4)`): Boxes for which the area will be computed. They are expected to be in (x1, y1, x2, y2) format with `0 <= x1 < x2` and `0 <= y1 < y2`. Returns: `torch.FloatTensor`: a tensor containing the area for each box. Nrrr)r\)r]s r'box_arear`osB ENE !Q$K%1+ %%1+ad *C DDr)c^t|}t|}tj|dddddf|ddddf}tj|dddddf|ddddf}||z j d}|dddddf|dddddfz}|dddf|z|z }||z } | |fS)Nr_rminr)r`r$maxrcclamp) boxes1boxes2area1area2left_top right_bottom width_heightinterunionious r'box_iourps V E V Eyy4! ,fQUm.rDrErFrIs`r'r>z$OwlViTObjectDetectionOutput.to_tuplerJr))rKrLrMrNr2r$rOrPr{dictrr|r5r6r}r7rr8rGrr>rQr)r'rzrzs8&*D%  d ")!Itd{!'+FE   $++/J!!D(/,0K""T)0-1L%##d*1-1L%##d*148186:3: %* r)rzzM Output type of [`OwlViTForObjectDetection.image_guided_detection`]. c0eZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ eed <dZeed <d eefd Zy) &OwlViTImageGuidedObjectDetectionOutputa logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`): Classification logits (including no-object) for all queries. image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. query_image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. target_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual target image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. query_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual query image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Class embeddings of all image patches. OWL-ViT represents images as a set of image patches where the total number of patches is (image_size / patch_size)**2. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nrr6query_image_embedstarget_pred_boxesquery_pred_boxesr}r7r8rcHtfdjDS)Nc3dK|]'}|dvr|nt|j)ywr;r<r?s r'rCzBOwlViTImageGuidedObjectDetectionOutput.to_tuple.. rDrErFrIs`r'r>z/OwlViTImageGuidedObjectDetectionOutput.to_tuple rJr))rKrLrMrNrr$rOrPr6rrrr}r7rr8rGrr>rQr)r'rrs8(,FE   $+-1L%##d*137))D0726u((4/615e''$.5-1L%##d*148186:3: %* r)rceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) OwlViTVisionEmbeddingsconfigct||j|_||_|j|_t jtj|j|_ t j|j|j |j|jd|_ |j|jzdz|_|jdz|_t j"|j |j |_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestridebiasr_r position_idsr persistent)super__init__ patch_sizer hidden_size embed_dimr Parameterr$randnclass_embeddingConv2d num_channelspatch_embedding image_size num_patches num_positions Embeddingposition_embeddingregister_bufferr%expandrBr __class__s r'rzOwlViTVisionEmbeddings.__init__s  ++ ++!||EKK8J8J,KL!yy++))$$  #--1B1BBqH!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr) embeddingsheightwidthrc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrg?rr_bicubicF)sizemode align_cornersdim)shaperweight unsqueezer$jit is_tracingrrrreshapepermuterr" interpolateviewcat) rBrrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r'interpolate_pos_encodingz/OwlViTVisionEmbeddings.interpolate_pos_encoding&sv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4  *11!Q1=BB1b#Nyy/?;CCr) pixel_valuesrch|j\}}}}|j|}|jdjdd}|jj |dd}t j||gd} |r| |j| ||z} | S| |j|jz} | S)Nr_rrr) rrflatten transposerrr$rrrr) rBrr batch_size_rr patch_embedsr}rs r'forwardzOwlViTVisionEmbeddings.forwardLs'3'9'9$ Avu++L9 #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr)F) rKrLrMrrr$rr[rrOboolr __classcell__rs@r'rrsmq1q*$D5<<$D$DUX$D]b]i]i$DL E$5$5 QU bgbnbn r)rc eZdZdeffd Z d dej dzdej dzdejdzdejfdZ xZ S) OwlViTTextEmbeddingsrc^t|tj|j|j |_tj|j|j |_|jdtj|jjddy)NrrFr) rrrr vocab_sizertoken_embeddingmax_position_embeddingsrrr$r%rrs r'rzOwlViTTextEmbeddings.__init__Zs !||F,=,=v?Q?QR"$,,v/M/MvOaOa"b  ELL)G)GHOOPWXej  r)N input_idsr inputs_embedsrc||jdn|jd}||jddd|f}||j|}|j|}||z}|S)Nr)rrrr)rBrrr seq_lengthposition_embeddingsrs r'rzOwlViTTextEmbeddings.forwardds{ -6,AY__R(}GZGZ[]G^  ,,Q  ^> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r)tensorseq_lenbszc|j|||j|jjddj S)Nrr_)rrrr contiguous)rBrrrs r'_shapezOwlViTAttention._shapes7{{3GQQRSUVWbbddr)N hidden_statesattention_maskcausal_attention_maskoutput_attentionsrc|j\}}}|j||jz}|j|j |d|} |j|j |d|} ||j zd|jf} |j|||j| }| j| } | j| } | jd} tj|| jdd} | j||j z|| fk7r/td||j z|| fd| j|{|j|d|| fk7r#td|d|| fd|j| j||j || |z} | j||j z|| } |{|j|d|| fk7r#td|d|| fd|j| j||j || |z} | j||j z|| } tjj| d} |r?| j||j || }|j||j z|| } nd}tjj!| |j |j" }|j%| j&}tj|| }|j||j z||jfk7r7td ||j ||jfd|j|j||j ||j}|jdd}|j)|||}|j+|}||fS) z#Input shape: Batch x Time x Channelrrr_z$Attention weights should be of size z , but is Nz!Attention mask should be of size r)ptrainingz `attn_output` should be of size )rrrrrrrrrr$bmmrrsrr"softmaxrrtorUrr)rBrrrrrtgt_lenr query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r'rzOwlViTAttention.forwards#0"4"4"6Wi{{=1DJJ> [[]!;RE {{4;;}#=r3G DNN*B > Ct{{<#>CCZP $Z__j1 (|((*5 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*  ! ,$))+Q/II 7a'8R7ST-22457(,,S$..'7SVkkL',,S4>>-A7GTL  %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<j|j@dz|zt j |jBj|jDdz|zt jF|jH|jjJnTt|tLrDt j|jN|jQ|jR|jTt|tVjXr>t jZ|j\t j^|jt|tVj`rOt j |jd||j\ t jZ|j\y y y ) zInitialize the weightsg{Gz?)meanstdrrr)r(r_N)1rinitializer_factor isinstancerinitnormal_rrrcopy_rr$r%rrrrrrinitializer_rangernum_hidden_layersrrrrrrr r OwlViTModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dim constant_ logit_scalelogit_scale_init_valueOwlViTForObjectDetectionbox_biascompute_box_biasnum_patches_heightnum_patches_widthrrzeros_rones_r)rBr$factor in_proj_std out_proj_stdfc_stds r' _init_weightsz#OwlViTPreTrainedModel._init_weights.s// f2 3 LL//66Sftm T LL2299RV W JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  6 7 LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d