CLIP variants

We’re on a journey to advance and democratize artificial intelligence through open source and open science. Back to Articles SigLIP 2: A better multilingual vision language encoder Published February 21, 2025 Update on GitHub Upvote 228 +222 Aritra Roy Gosthipaty ariG23498 Follow merve merve Follow Pavel Iakubovskii qubvel-hf Follow TL;DR Today Google releases a new and better family of multilingual vision-language encoders, SigLIP 2. The authors have extended the training objective of SigLIP (sigmoid loss) with additional objectives for improved semantic understanding, localization, and dense features. Additional objectives (Source: https://huggingface.co/papers/2502.14786 ) SigLIP 2 models outperform the older SigLIP ones at all model scales in core capabilities, including zero-shot classification, image-text retrieval, and transfer performance when extracting visual representations for Vision-Language Models (VLMs). A cherry on top is the dynamic resolution (naflex) variant. This is useful for downstream tasks sensitive to aspect ratio and resolution. Here is a list of all the models released: Size Patch Size Resolution Transformers JAX Base (86M) 32 256 google/siglip2-base-patch32-256 google/siglip2-base-patch32-256-jax 16 224 google/siglip2-base-patch16-224 google/siglip2-base-patch16-224-jax 256 google/siglip2-base-patch16-256 google/siglip2-base-patch16-256-jax 384 google/siglip2-base-patch16-384 google/siglip2-base-patch16-384-jax 512 google/siglip2-base-patch16-512 google/siglip2-base-patch16-512-jax - - google/siglip2-base-patch16-naflex google/siglip2-base-patch16-naflex-jax Large (303M) 16 256 google/siglip2-large-patch16-256 google/siglip2-large-patch16-256-jax 384 google/siglip2-large-patch16-384 google/siglip2-large-patch16-384-jax 512 google/siglip2-large-patch16-512 google/siglip2-large-patch16-512-jax Shape Optimized 400M 14 224 google/siglip2-so400m-patch14-224 google/siglip2-so400m-patch14-224-jax 384 google/siglip2-so400m-patch14-384 google/siglip2-so400m-patch14-384-jax 16 256 google/siglip2-so400m-patch16-256 google/siglip2-so400m-patch16-256-jax 384 google/siglip2-so400m-patch16-384 google/siglip2-so400m-patch16-384-jax 512 google/siglip2-so400m-patch16-512 google/siglip2-so400m-patch16-512 - - google/siglip2-so400m-patch16-naflex google/siglip2-so400m-patch16-naflex-jax Giant (1B) 16 256 google/siglip2-giant-opt-patch16-256 google/siglip2-giant-opt-patch16-256-jax 384 google/siglip2-giant-opt-patch16-384 google/siglip2-giant-opt-patch16-384-jax Introduction Vision encoders are simple - they take an image, encode it into a representation, and that representation is used for downstream tasks like classification, object detection, image segmentation, and more vision tasks. Researchers are always in pursuit of visual representations that are dense, locality-aware, and semantically rich. CLIP and ALIGN are the first examples of image encoders and text encoders aligned together through joint training. This approach opened new ways to train vision models. SigLIP took it further, replacing CLIP's contrastive loss with sigmoid loss for even better encoders. The takeaway? With smarter training objectives, we keep building vision encoders that are more structured, fine-grained, and powerful. SigLIP 2 is just that, a bunch of really interesting and smart training objectives applied on top of that of SigLIP's to provide better and stronger vision language encoders. We will try something new with this blog post. Rather than stating what is new and where to find it, we will go through a little exercise together. We start off with SigLIP and then brainstorm a series of questions (prefixed with 🤔) and answers (a new heading) to gradually cover all the updates in SigLIP 2. Sounds good? We will begin our journey with the vision encoder where the patch size is 16, and the image resolution is 256. We have four variants to start our training: siglip2-base-patch16-256 siglip2-large-patch16-256 siglip2-so400m-patch16-256 siglip2-giant-opt-patch16-256 🤔 Question 1: What is a (low effort) auxiliary training objective that we can use to learn better visual representations (in terms of location awareness and sense of locality)? Add a decoder (it’s that simple) Let’s add a decoder to the mix. Now we have an image encoder, a text encoder, and a text decoder. The text decoder will have three objectives: Predict a holistic image caption Predict bounding box coordinates given captions describing specific image regions Predict region-specific caption given bounding box coordinates The decoder provides an additional signal to the vision encoder, making it location-aware. This marks the first improvement to the training recipe in SigLIP 2. 🤔 Question 2: How do we improve fine-grained local semantics of the image representation? Self-distillation with Global-Local loss and Masked Prediction To improve fine-grained local semantics in image representation, we introduce two key training objectives, Global-Local Loss, and Masked Prediction Loss. Taking inspiration from self-supervised learning literature, we use self-distillation. We can use a model as a teacher, and the same model as a student. Upon each iteration the teacher will be the moving average of the student's parameters. Global-Local Loss: The student network gets a partial (local) view of the training image, and is trained to match the teacher’s representation, derived from the full image. Masked Prediction Loss: 50% of the embedded image patches in the student network are masked with mask tokens. The student needs to match the features of the teacher at masked locations. These objectives teach the vision encoder to be spatially aware and improve its local semantics. The authors add this loss only after 80% of the training is done with the sigmoid and decoder loss. This is done in order to save compute (additional losses are pretty expensive) and to not negatively affect the encoders. 🤔 Question 3: How to adapt models to different resolutions? Adapting to different resolutions It is a known fact that image models can be very sensitive to varying resolutions and aspect ratios. Here we can leverage two distinct methodologies to adapt these models on different resolutions and patch sizes. Fixed resolution variant: Taking the checkpoints from 95% training, we can resize the positional embeddings and the patch embeddings and then continue training for a requested (potentially larger) resolution. Dynamic resolution variant: Taking inspiration from FlexiViT, which uses inputs with different sequence lengths, and NaViT, which adheres to the native aspect ratios, we can create NaFlex variants. This is interesting because we can use a single model for OCR (little aspect ratio distortion) and document understanding (appropriate resolution). Models with the -naflex suffix are the dynamic resolution variants. While the fixed-resolution models can be used out of the box with the existing SiglipModel class, you would need to use Siglip2Model to use the naflex variants. We handle this automatically when you use the pipeline API! This brings us to the end of the evolution from SigLIP to SigLIP 2. In the next sections we will look at applications with SigLIP 2. Run inference with transformers Running inference on the models is pretty straightforward. You can copy paste the code below and run inference on a free tier Colab notebook 🚀 To run inference on SigLIP 2, please install transformers from main or from this stable branch: pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2 Zero-shot Classification Here we use the handy pipeline API to showcase zero-shot classification capabilities for SigLIP 2. from transformers import pipeline ckpt = "google/siglip2-so400m-patch14-384" pipe = pipeline(model=ckpt, task="zero-shot-image-classification") inputs = { "images": [ "https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000285.jpg", # bear "https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000776.jpg", # teddy bear ], "texts": [ "bear looking into the camera", "bear looking away from the camera", "a bunch of teddy bears", "two teddy bears", "three teddy bears" ], } outputs = pipe(inputs["images"], candidate_labels=inputs["texts"]) Let’s visualize the outputs. Zero Shot Classification Scores Visualized Encode images for downstream tasks You can also encode images using the following: import torch from transformers import AutoModel, AutoProcessor from transformers.image_utils import load_image ckpt = "google/siglip2-so400m-patch14-384" model = AutoModel.from_pretrained(ckpt, device_map="auto").eval() processor = AutoProcessor.from_pretrained(ckpt) image = load_image("https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000285.jpg") inputs = processor(images=[image], return_tensors="pt").to(model.device) with torch.no_grad(): image_embeddings = model.get_image_features(**inputs) print(image_embeddings.shape) # torch.Size([1, 1152]) Comparing SigLIP 1 with SigLIP 2 Looking at the table of all the SigLIP 2 models released, we see two distinct changes from SigLIP: SigLIP 2 has new variants (naflex) for dynamic resolution. SigLIP 2 adds a giant (1B) series. The evaluation table of SigLIP 2 demonstrates its superiority over SigLIP. Evaluation Scores for SigLIP 2 (Source: https://huggingface.co/papers/2502.14786 ) Here is a demo where one can compare the zero-shot classification results of SigLIP 1 and SigLIP 2. Using the encoder for VLMs Vision encoders aligned to textual information have become increasingly vital in the development of Vision Language Models (VLMs). A common approach to building VLMs involves combining a pretrained vision encoder with a pretrained LLM, and training them together using multimodal data across a diverse set of vision-language tasks. One standout example of a VLM leveraging the SigLIP family of vision encoders is PaliGemma. One can dive deeper into PaliGemma's capabilities in this PaliGemma blog post. Building on this foundation, the recently introduced PaliGemma 2 takes it a step further by integrating SigLIP with the advanced Gemma 2 LLM. It would be really exciting to swap out SigLIP with SigLIP 2 in a PaliGemma like setting and see how that model fares. Acknowledgements We would like to thank Michael Tschannen (first author of SigLIP 2), Vaibhav Srivastav and Sayak Paul for feedback on this blog post. A huge shout out to the Google team for releasing this amazing, and open, model family. In no particular order we would like to thank Pavel, Ross, Pablo, Pedro, Lysandre and the rest of the Hugging Face team for their immense support and contribution towards this project. Models mentioned in this article 10 Papers mentioned in this article 3 Collections mentioned in this article 1 More Articles from our Blog multimodalnlpcommunity Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers 82 April 16, 2026 multimodalnlpcommunity Multimodal Embedding & Reranker Models with Sentence Transformers 75 April 9, 2026 Community drscotthawley Feb 21, 2025 • edited Feb 21, 2025 Might want to check something: The demo at the bottom hangs with "Uploading 1 file..." and none of the premade examples load or execute. 6 different Javascript console errors are thrown. (Chrome, Mac). See translation 2 replies · ➕ 2 2 + theaussiestew Feb 21, 2025 Yes I see this too Expand 1 reply rwightman Feb 21, 2025 OpenCLIP & timm weights are up now, https://huggingface.co/collections/timm/siglip-2-67b8e72ba08b09dd97aecaf9 See translation 🚀 7 7 + Reply birdortyedi Feb 22, 2025 Thanks for blog. Siglip2 does not include 'spiece.model' as in Siglip. Is this expected? The processor raises an error indicating that the vocab file is a NoneType object, which may suggest a missing spiece model. Any ideas? transformers.version : 4.49.0 See translati

Deploy SigLIP-2, JinaCLIP-v2, and Cohere Embed-v4 for vision-text RAG on GPU cloud. VRAM sizing, Infinity-Embedding setup, cost vs managed APIs, and a full production deployment guide. At 500M image-text pairs per month, Cohere Embed-v4's managed API costs roughly $50,000. A single A100 80GB on Spheron at $1.19/hr spot pricing processes the same volume for roughly $9,000-10,000. That gap is why teams building product search, visual knowledge bases, and mixed-media RAG pipelines are moving multimodal embedding inference in-house. This guide covers production deployment of four multimodal embedding models: SigLIP-2, JinaCLIP-v2, Cohere Embed-v4, and Voyage-multimodal-3. You'll get VRAM sizing tables with live-fetched Spheron pricing, a complete Infinity-Embedding setup, vector database integration patterns, and a cost-per-million benchmark you can use to find your own crossover point. For text-only embedding deployment, the TEI self-hosting guide covers the text-side stack with BGE-M3, Qwen3-Embedding, and cross-encoder rerankers. Why Text-Only Embeddings Break for Visual Knowledge Bases A text embedding model encodes the words you feed it. That sounds obvious, but it becomes a real problem once your document corpus includes anything that does not translate cleanly to text. Three failure modes come up repeatedly in production: Charts and graphs. A revenue chart has axes, legend labels, and data points. When you flatten it to text with OCR or a caption model, you get something like "Q1 Q2 Q3 Q4 Revenue Growth YoY". The structural relationships that make the chart meaningful, which bar is tallest, what the trend is, are gone. A multimodal embedding model sees the image directly and preserves that structure in the vector. Screenshot-heavy UIs. Product documentation that ships as screenshots has no parsable text at all for most of its information surface. Error dialogs, UI state captures, and configuration screenshots are all noise if you rely on text extraction. A CLIP-family model can retrieve the screenshot of the error dialog when the user asks about that error. Product catalog images. A red mesh running shoe has shape, material texture, color, and silhouette. The text description might say "athletic shoe, red" and miss the visual details that distinguish it from ten other red shoes. Multimodal embeddings capture those visual features directly. Content type Text embedding score Multimodal embedding score Gap Charts and data visualizations Low (axis text only) High (structure preserved) Large UI screenshots and error dialogs Very low (no OCR text) Medium-high Very large Product images with visual detail Low-medium (metadata only) High Large Scanned document text pages Medium (OCR dependent) Medium Small Text-heavy slides with bullet points High High Minimal The gap matters most when your corpus is more than 20-30% non-text content. Below that threshold, text-only embeddings with good OCR preprocessing are usually adequate, and for that scanned-document tier the DeepSeek-OCR deployment guide covers self-hosting a dedicated OCR pass that outperforms flattening images to text with a generic caption model. The 2026 Multimodal Embedding Landscape Five models are worth evaluating for production vision-text RAG as of June 2026: Model Architecture Output dim Max image resolution Open weights Serving recommendation SigLIP-2 ViT-SO400M SigLIP dual-encoder (ViT-SO400M + text) 1152 512x512 Yes (Apache 2.0) Infinity-Embedding JinaCLIP-v2 CLIP dual-encoder (ViT-L/14 + Jina text) 1024 (Matryoshka: 64-1024) 224x224 Yes (Apache 2.0) Infinity-Embedding Cohere Embed-v4 Proprietary 1024 Variable Commercial license tier vLLM (self-hosted) or API Voyage-multimodal-3 Proprietary dual-encoder 1024 Variable No (API only) Managed API only NV-Embed-Multimodal NVIDIA ViT + text encoder 4096 336x336 Yes (research license) Triton or TGI Benchmark scores (MTEB-Multimodal and Vidore-v2 leaderboard positions) as of June 2026. Scores update as new model versions are submitted. SigLIP-2 ViT-SO400M is currently the strongest open model for pure image-text similarity tasks. Google trained it with a sigmoid loss function rather than the standard InfoNCE contrastive loss, which produces better-calibrated similarity scores and stronger performance on fine-grained image retrieval benchmarks. The SO400M variant (400M parameter ViT-SO backbone) requires more VRAM than smaller SigLIP-2 variants but delivers meaningfully better retrieval accuracy on visual content. JinaCLIP-v2 adds two things SigLIP-2 lacks: multilingual text support across 89 languages and Matryoshka Representation Learning. The Matryoshka option lets you shrink embeddings from 1024 to 64 dimensions at serving time with modest accuracy loss, which matters when you are storing millions of embeddings and every byte counts. Use JinaCLIP-v2 when your corpus has non-English text alongside images. Cohere Embed-v4 is the strongest performer on MTEB-Multimodal benchmarks as of Q2 2026, but the self-hostable weights tier requires a commercial license agreement with Cohere separate from the API terms. Do not assume the weights are freely redistributable. The API mode is the path of least resistance for low-volume workloads; self-hosting makes sense above 200-300M tokens per month where the cost math clearly favors it. Voyage-multimodal-3 from Voyage AI is API-only. Voyage does not publish self-hostable weights for this model. It appears in this comparison for teams evaluating managed API options but the deployment walkthrough sections below do not cover it. NV-Embed-Multimodal from NVIDIA Research delivers the highest output dimensionality (4096-dim) and is designed for retrieval tasks where maximum discriminability matters. The research license restricts commercial use; confirm terms before deploying in production. For ColPali-style late-interaction document retrieval (PDFs, slides) where you need patch-level visual matching rather than single-vector dense search, see ColPali on GPU Cloud as an alternative to the dense multimodal embeddings covered here. Hardware Sizing The following table uses live-fetched pricing from the Spheron API as of June 2, 2026. GPU VRAM SigLIP-2 batch size JinaCLIP-v2 batch size Throughput (img-text pairs/hr) On-demand $/hr Spot $/hr L40S 48GB 128 256 ~40,000-60,000 $0.96 N/A A100 80GB PCIe 80GB 256 512 ~55,000-75,000 $1.48 $1.19 H100 SXM5 80GB 256 512 ~75,000-95,000 $3.92 $2.91 B200 SXM6 192GB 512 1024 ~120,000-160,000 $7.35 $2.68 Throughput figures are at FP16 precision, batch size as listed, with Infinity-Embedding and CUDA graph optimization enabled. Image preprocessing (resize, normalize) runs on-GPU via the model's AutoProcessor. Exact throughput varies based on image resolution, JPEG decode overhead, and whether text and image batches run concurrently; treat these as estimates for planning purposes. Pricing fluctuates based on GPU availability. The prices above are based on 02 Jun 2026 and may have changed. Check current GPU pricing for live rates. L40S 48GB is the production sweet spot for embedding-only deployments. 48GB GDDR6 provides enough VRAM to run multiple model workers or a model plus a co-located reranker. At $0.96/hr, it offers a strong cost-per-pair ratio for mid-volume workloads. L40S on Spheron A100 80GB PCIe is the best pick for pure embedding throughput per dollar. The A100's HBM2e memory bandwidth (1.94 TB/s) handles large batches efficiently and spot pricing at $1.19/hr makes it cheaper than the L40S for high-utilization workloads. A100 on Spheron H100 SXM5 makes sense when you are co-locating the embedding server with a 7B-14B LLM (see LLM serving optimization guide ) in the same RAG stack, where the combined VRAM budget requires 80GB. The H100's 3.35 TB/s HBM3 bandwidth delivers noticeably higher throughput than the A100 on large batch sizes but is overkill for embedding-only workloads. H100 on Spheron B200 SXM6 is for teams running very high-volume indexing jobs (10M+ pairs per day) or co-locating embedding plus a 70B+ LLM. 192GB HBM3e handles multiple concurrent model replicas without VRAM pressure. Spot pricing at $2.68/hr makes it cost-competitive with H100 on-demand for sustained workloads. Production Deployment Infinity-Embedding Setup Infinity-Embedding (github.com/michaelfeil/infinity) is the recommended serving layer for CLIP-family models. It ships with OpenAI-compatible /embeddings, /v1/embeddings, and model-specific /encode_image and /encode_text endpoints, and handles image preprocessing (resize, normalize) automatically using each model's registered AutoProcessor. Deploy JinaCLIP-v2: bash docker run --gpus all \ -p 7997:7997 \ -e INFINITY_LOG_LEVEL=INFO \ -e INFINITY_BATCH_SIZE=128 \ -e INFINITY_DEVICE=cuda \ michaelfeil/infinity:latest v2 \ --model-id jinaai/jina-clip-v2 \ --served-model-name jina-clip-v2 \ --batch-size 128 \ --device cuda Deploy SigLIP-2 ViT-SO400M: bash docker run --gpus all \ -p 7997:7997 \ -e INFINITY_LOG_LEVEL=INFO \ -e INFINITY_BATCH_SIZE=128 \ -e INFINITY_DEVICE=cuda \ michaelfeil/infinity:latest v2 \ --model-id google/siglip2-so400m-patch16-512 \ --served-model-name siglip2 \ --batch-size 128 \ --device cuda The server starts downloading model weights on first run. On an A100 80GB, model load takes 30-60 seconds for SigLIP-2 SO400M. Set --preload-only to warm the model before opening the port if you need zero cold-start latency on the first request. Embed an image via base64: bash IMAGE_B64=$(base64 -w 0 /path/to/your/image.jpg) curl -s http://localhost:7997/v1/embeddings \ -H "Content-Type: application/json" \ -d "{ \"model\": \"jina-clip-v2\", \"input\": [\"data:image/jpeg;base64,${IMAGE_B64}\"] }" | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d['data'][0]['embedding']), 'dims')" Always use base64-encoded images in production API calls rather than image URLs. URL-based inputs add latency proportional to the image download time and introduce a network dependency that can cause timeout failures under load. Embed text: bash curl -s http://localhost:7997/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "model": "jina-clip-v2", "input": ["a photo of a red running shoe"] }' Cohere Embed-v4 Deployment Cohere Embed-v4 does not publish weights on Hugging Face for general use. The self-hosted weights tier requires a direct commercial agreement with Cohere, and there is no publicly documented vLLM model registration for cohere/embed-v4 accessible without Cohere customer credentials. For most users, Cohere Embed-v4 is API-only. Use the Cohere Python client (cohere.Client) to access it via the managed API. Contact Cohere directly if you need self-hosted weights for a production deployment. The API is the correct path for low-to-medium volume workloads; at high volumes (200M+ tokens/month), evaluate whether Cohere's enterprise self-hosting terms make economic sense for your use case. For the generation step in a full Cohere-native RAG pipeline, Command A on GPU Cloud is the natural pairing. 111B parameters, 256K context, and native compatibility with Cohere's embed and rerank APIs, all running on your own GPU infrastructure. Image Preprocessing Pipeline Each model has specific preprocessing requirements. Using the wrong normalization is one of the most common causes of low retrieval accuracy when switching between models. Model Normalization mean Normalization std Input resolution SigLIP-2 (all variants) [0.5, 0.5, 0.5] [0.5, 0.5, 0.5] 512x512 JinaCLIP-v2 [0.485, 0.456, 0.406] [0.229, 0.224, 0.225] 224x224 NV-Embed-Multimodal [0.48145466, 0.4578275, 0.40821073] [0.26862954, 0.26130258, 0.27577711] 336x336 When using Infinity-Embedding via the API, the server handles preprocessing automatically. When embedding directly via Hugging Face Transformers, always load the processor from the model's repo: python from transformers import AutoProcessor, AutoModel import torch from PIL import I

SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features Michael Tschannen Affiliation: Core contributor Alexey Gritsenko Affiliation: Core contributor Xiao Wang Affiliation: Core contributor Muhammad Ferjad Naeem Affiliation: Core contributor Ibrahim Alabdulmohsin Affiliation: Core contributor Nikhil Parthasarathy Affiliation: Core contributor Talfan Evans Affiliation: Core contributor Lucas Beyer Affiliation: Core contributor Ye Xia Basil Mustafa Olivier Hénaff Jeremiah Harmsen Andreas Steiner Xiaohua Zhai Affiliation: Google DeepMind Affiliation: Core contributor Affiliation: Project lead Affiliation: Work done while at Google DeepMind Abstract We introduce SigLIP 2, a family of new multilingual vision-language encoders that build on the success of the original SigLIP. In this second iteration, we extend the original image-text training objective with several prior, independently developed techniques into a unified recipe—this includes captioning-based pretraining, self-supervised losses (self-distillation, masked prediction) and online data curation. With these changes, SigLIP 2 models outperform their SigLIP counterparts at all model scales in core capabilities, including zero-shot classification, image-text retrieval, and transfer performance when extracting visual representations for Vision-Language Models (VLMs). Furthermore, the new training recipe leads to significant improvements on localization and dense prediction tasks. We also train variants which support multiple resolutions and preserve the input’s native aspect ratio. Finally, we train on a more diverse data-mixture that includes de-biasing techniques, leading to much better multilingual understanding and improved fairness. To allow users to trade off inference cost with performance, we release model checkpoints at four sizes: ViT-B (86M), L (303M), So400m (400M), and g (1B). 1 Introduction Contrastive image-text embedding models trained on billion-scale datasets, as pioneered by CLIP [ 50 ] and ALIGN [ 28 ], have become the mainstream approach for high-level, semantic understanding of visual data. These models enable fine-grained, zero-shot classification rivaling the quality of supervised methods and enable efficient text-to-image and image-to-text retrieval. Furthermore, they lead to excellent vision-language understanding capabilities when combined with Large Language Models (LLMs) to build Vision-Language Models (VLMs). Developing on the success of CLIP, several improvements have been proposed such as re-captioning images [ 38 ], adding image-only self-supervised losses [ 45, 38 ], and training with a small decoder for auxiliary tasks such as captioning and localization [ 67, 32, 62 ]. At the same time, several groups have released model checkpoints for the open-source community [ 50, 70, 27, 57, 19 ]. However, these releases do not include the full breadth of latest improvements into a single model, as they all relatively closely follow CLIP’s original approach. Here, building on the SigLIP training recipe [ 71 ], we incorporate several improvements from prior work and release a new family of open models11 1 Model checkpoints are available at https://github.com/google-research/big_vision/tree/main/ big_vision/configs/proj/image_text/README_siglip2.md that both excel on CLIP’s core capabilities—–zero-shot classification, retrieval, and feature extraction for VLMs—and improve areas where vanilla CLIP-style models lag behind, including localization and extracting dense, semantic representations. In summary, SigLIP 2 models provide the following: • Strong multilingual vision-language encoders: SigLIP 2 shows excellent performance on English-focused vision-language tasks while providing strong results on multilingual benchmarks with a single model. This enables use in a wide range of languages and cultural contexts. • Dense features: We incorporate self-supervised losses as well as a decoder-based loss, which result in better dense features (e.g. for segmentation and depth estimation) and improve localization tasks (such as referring expression comprehension). • Backward compatibility: SigLIP 2 is designed to be backward compatible with SigLIP by relying on the same architecture. This allows existing users to simply swap out the model weights and tokenizer (which is now multilingual) to get improvements on a wide range of tasks. • Native aspect ratio and variable resolution: SigLIP 2 also includes a NaFlex variant, which supports multiple resolutions and preserves the native image aspect ratio. These models have the potential to improve aspect sensitive applications such as document understanding. • Strong small models: SigLIP 2 further optimizes performance of smaller models (B/16 and B/32 models), by using techniques in distillation via active data curation. In the next section we provide a detailed description of the SigLIP 2 training recipe. Sec. 3 presents evaluations of SigLIP 2 models and baselines across a variety of tasks and benchmarks. Finally, Sec. 4 gives a short overview of related work, and conclusions can be found in Sec. 5. Figure 1: SigLIP 2 adds the captioning-based pretraining from LocCa [ 62 ] as well as self-distillation and masked prediction from SILC [ 45 ] and TIPS [ 38 ] (during the last 20% of training) to the sigmoid loss from SigLIP [ 71 ]. For some variants, the recipe additionally involves fine-tuning with data curation [ 61 ] or adaptation to native aspect ratio and variable sequence length [ 6, 12 ]. 2 Training recipe We combine the original SigLIP training recipe [ 71 ] with decoder-based pretraining [ 60, 62 ], in addition to self-distillation and masked prediction as in the DINO line of work [ 9, 47 ] (see Fig. 1 for an overview). Pretraining an image encoder with a language decoder for captioning and referring expression comprehension was shown to improve OCR capabilities and localization [ 62 ], whereas self-distillation and masked prediction leads to better features for dense prediction tasks, zero-shot classification and retrieval [ 45, 38 ]. Rather than combining all these techniques in a single run we follow a staged approach as outlined below to manage the computational and memory overhead compared to SigLIP training. In addition to training a set of models and adapting each model separately to different resolutions while distorting the aspect ratio, we also train variants which process images while largely preserving their native aspect ratio like NaViT [ 12 ] and support different sequence lengths as FlexiViT [ 6 ]. We call this variant NaFlex, described in Sec. 2.4.2. Finally, to improve the quality of the smallest models we fine-tune those with implicit distillation via active sample selection, following the approach from [ 61 ]. 2.1 Architecture, training data, optimizer For the architecture, we follow SigLIP [ 71 ] so that existing users can simply swap out the encoder weights. Specifically, the fixed-resolution variant relies on the standard ViT architecture [ 15 ] with learned positional embedding. We use the same architecture for the image and text tower, except for the g-sized vision encoder which is paired with an So400m-sized [ 1 ] text encoder. Vision and text representations are pooled using a MAP head (attention pooling) [ 69 ]. We set the text length to 64 and use the multilingual Gemma tokenizer [ 22 ] with vocabulary size 256k, transforming the text to lower case before tokenization. We use the WebLI dataset [ 10 ] containing 10 billion images and 12 billion alt-texts covering 109 languages. To strike a good balance between quality on English and multilingual vision-language benchmarks we compose the mixture such that 90% of the training image-text pairs is sourced from English web pages, and the remaining 10% from non-English web pages, as recommended in [ 49 ]. We further apply the filtering techniques from [ 2 ] to mitigate data biases in representation and association with respect to sensitive attributes. Unless noted otherwise, we use the Adam optimizer with learning rate 10−310^{-3}, decoupled weight decay 10−410^{-4} [ 37 ], and gradient clipping to norm 1. We set the batch size to 32k and use a cosine schedule with 20k warmup steps, training for a total of 40B examples. Our models are trained on up to 2048 TPUv5e chips [ 24 ] using a fully-sharded data-parallel strategy (FSDP [ 72 ]). 2.2 Training with Sigmoid loss and decoder In the first step of pretraining, we combine SigLIP [ 71 ] with LocCa [ 62 ] by simply combining the two losses with equal weight. Unlike CLIP [ 50 ], which relies on a contrastive loss, SigLIP creates binary classification problems by combining every image embedding with every text embedding in the mini-batch and trains the embeddings to classify matching and non-matching pairs via logistic regression (sigmoid loss). We use the original implementation and refer to [ 71 ] for details. For LocCa, we attach a standard transformer decoder with cross-attention to the un-pooled vision encoder representation (before applying the MAP head). The decoder follows the shapes of the text encoder except that we add cross-attention layers and reduce the number of layers by a factor of two. Besides image captioning, LocCa also trains for automatic referring expression prediction and grounded captioning. The former amounts to predicting bounding box coordinates for captions describing specific image regions, whereas the latter involves predicting region-specific captions given bounding box coordinates. Region-caption pairs are automatically annotated by first extracting n-grams from the alt-texts and then applying open-vocabulary detection using the recipe from [ 41 ]. Additionally, we use the fixed set of object categories from [ 10 ] instead of n-grams. For each example, the decoder is trained to predict all three targets (amounting to three decoder forward-passes). The captioning target is predicted with parallel prediction [ 60 ] with probability of 50%, i.e. all caption tokens are predicted in parallel from mask tokens, without causal attention mask. Please refer to [ 62 ] for more detail. Finally, to reduce memory consumption due to the large vocabulary, we implement a chunked version of the decoder loss. For all model sizes, we set the vision encoder patch size to 16 and the image resolution to 256 (resulting in an image representation sequence length of 256). Finally, we note that the decoder only serves for representation learning here and is not part of the model release. 2.3 Training with self-distillation and masked prediction Following SILC [ 45 ] and TIPS [ 38 ], we augment the training setup described in Sec. 2.2 with local-to-global correspondence learning with self-distillation and masked prediction losses [ 9, 75, 47 ] to improve the local semantics of the (un-pooled) feature representation. This representation is typically used for dense prediction tasks like segmentation, depth estimation etc. Concretely, we add two terms to the losses described in Sec. 2.2 as detailed next. The first term is the local-to-global consistency loss from [ 45 ], in which the vision encoder becomes the student network, which gets a partial (local) view of the training image, and is trained to match the teacher’s representation, derived from the full image. This auxiliary matching task is performed in a high-dimensional feature space computed with a separate MLP head. As is common in the literature, the teacher parameters are obtained as an exponential moving average (EMA) of the student parameters over the previous iterations. We rely on a single global (teacher) view and 8 local (student) views and otherwise follow the augmentations, loss and hyper parameters from [ 45 ]. The second loss term is the masked prediction objective from [ 38 ]. We replace 50% of the embedded image patches in the student network

We’re on a journey to advance and democratize artificial intelligence through open source and open science. Transformers documentation SigLIP Transformers 🏡 View all docsAWS Trainium & InferentiaAccelerateArgillaAutoTrainBitsandbytesCLIChat UIDataset viewerDatasetsDeploying on AWSDiffusersDistilabelEvaluateGoogle CloudGoogle TPUsGradioHubHub Python LibraryHuggingface.jsInference Endpoints (dedicated)Inference ProvidersKernelsLeRobotLeaderboardsLightevalMicrosoft AzureOpenEnvOptimumPEFTReachy MiniSafetensorsSentence TransformersTRLTasksText Embeddings InferenceText Generation InferenceTokenizersTrackioTransformersTransformers.jsXetsmolagentstimm Search documentation mainv5.17.0v5.15.1v5.14.0v5.13.1v5.12.0v5.11.0v5.10.4v5.9.0v5.8.1v5.7.0v5.6.2v5.5.4v5.4.0v5.3.0v5.2.0v5.1.0v5.0.0v4.57.6v4.56.2v4.55.4v4.53.3v4.52.3v4.51.3v4.50.0v4.49.0v4.48.2v4.47.1v4.46.3v4.45.2v4.44.2v4.43.4v4.42.4v4.41.2v4.40.2v4.39.3v4.38.2v4.37.2v4.36.1v4.35.2v4.34.1v4.33.3v4.32.1v4.31.0v4.30.0v4.29.1v4.28.1v4.27.2v4.26.1v4.25.1v4.24.0v4.23.1v4.22.2v4.21.3v4.20.1v4.19.4v4.18.0v4.17.0v4.16.2v4.15.0v4.14.1v4.13.0v4.12.5v4.11.3v4.10.1v4.9.2v4.8.2v4.7.0v4.6.0v4.5.1v4.4.2v4.3.3v4.2.2v4.1.1v4.0.1v3.5.1v3.4.0v3.3.1v3.2.0v3.1.0v3.0.2v2.11.0v2.10.0v2.9.1v2.8.0v2.7.0v2.6.0v2.5.1v2.4.1v2.3.0v2.2.2v2.1.1v2.0.0v1.2.0v1.1.0v1.0.0doc-builder-html ARDEENESFRHIITJAKOPTROTRZH Join the Hugging Face community and get access to the augmented documentation experience Collaborate on models, datasets and Spaces Faster examples with accelerated inference Switch between documentation themes Sign Up to get started This model was published in HF papers on 2023-03-27 and contributed to Hugging Face Transformers on 2024-01-08. Copy page SigLIP SigLIP is a multimodal image-text model similar to CLIP. It uses separate image and text encoders to generate representations for both modalities. Unlike CLIP, SigLIP employs a pairwise sigmoid loss on image-text pairs during training. This training loss eliminates the need for a global view of all pairwise similarities between images and texts within a batch. Consequently, it enables more efficient scaling to larger batch sizes while also delivering superior performance with smaller batch sizes. You can find all the original SigLIP checkpoints under the SigLIP collection. Click on the SigLIP models in the right sidebar for more examples of how to apply SigLIP to different image and text tasks. The example below demonstrates how to generate similarity scores between texts and image(s) with Pipeline or the AutoModel class. Pipeline AutoModel Copied from transformers import pipeline image = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" candidate_labels = ["a Pallas cat", "a lion", "a Siberian tiger"] pipeline = pipeline(task="zero-shot-image-classification", model="google/siglip-base-patch16-224", device=0) pipeline(image, candidate_labels=candidate_labels) Quantization reduces the memory burden of large models by representing the weights in a lower precision. Refer to the Quantization overview for more available quantization backends. The example below uses bitsandbytes to only quantize the weights to int4. Copied import requests import torch from PIL import Image from transformers import AutoModel, AutoProcessor, BitsAndBytesConfig bnb_config = BitsAndBytesConfig(load_in_4bit=True) model = AutoModel.from_pretrained("google/siglip-base-patch16-224", quantization_config=bnb_config, device_map="auto", attn_implementation="sdpa") processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224") url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" image = Image.open(requests.get(url, stream=True).raw) candidate_labels = ["a Pallas cat", "a lion", "a Siberian tiger"] texts = [f'This is a photo of {label}.' for label in candidate_labels] inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model(**inputs) logits_per_image = outputs.logits_per_image probs = torch.sigmoid(logits_per_image) print(f"{probs[0][0]:.1%} that image 0 is '{candidate_labels[0]}'") Notes Training is supported for DDP and FSDP on single-node multi-GPU setups. However, it does not use torch.distributed utilities which may limit the scalability of batch size. When using the standalone SiglipTokenizer or SiglipProcessor, make sure to pass padding="max_length" because that is how the model was trained. To get the same results as the Pipeline, a prompt template of "This is a photo of {label}." should be passed to the processor. Toggle the attn_implementation parameter to either "sdpa" or "flash_attention_2" to use a more memory-efficient attention. Copied # pip install -U flash-attn --no-build-isolation from transformers import SiglipModel model = SiglipModel.from_pretrained( "google/siglip-so400m-patch14-384", attn_implementation="flash_attention_2", device_map="auto", ) SiglipConfig class transformers.SiglipConfig < source > ( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: typing.Union[str, ForwardRef('torch.dtype'), NoneType] = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: typing.Optional[typing.Literal['regression', 'single_label_classification', 'multi_label_classification']] = Nonetext_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Nonevision_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Noneinitializer_factor: float = 1.0 ) Parameters text_config (Union[dict, ~configuration_utils.PreTrainedConfig], optional) — The config object or dictionary of the text backbone. vision_config (Union[dict, ~configuration_utils.PreTrainedConfig], optional) — The config object or dictionary of the vision backbone. initializer_factor (float, optional, defaults to 1.0) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing). This is the configuration class to store the configuration of a SiglipModel. It is used to instantiate a Siglip model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the google/siglip-base-patch16-224 Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information. Example: Copied >>> from transformers import SiglipConfig, SiglipModel >>> # Initializing a SiglipConfig with google/siglip-base-patch16-224 style configuration >>> configuration = SiglipConfig() >>> # Initializing a SiglipModel (with random weights) from the google/siglip-base-patch16-224 style configuration >>> model = SiglipModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config >>> # We can also initialize a SiglipConfig from a SiglipTextConfig and a SiglipVisionConfig >>> from transformers import SiglipTextConfig, SiglipVisionConfig >>> # Initializing a SiglipText and SiglipVision configuration >>> config_text = SiglipTextConfig() >>> config_vision = SiglipVisionConfig() >>> config = SiglipConfig(text_config=config_text, vision_config=config_vision) SiglipTextConfig class transformers.SiglipTextConfig < source > ( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: typing.Union[str, ForwardRef('torch.dtype'), NoneType] = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: typing.Optional[typing.Literal['regression', 'single_label_classification', 'multi_label_classification']] = Nonevocab_size: int = 32000hidden_size: int = 768intermediate_size: int = 3072num_hidden_layers: int = 12num_attention_heads: int = 12max_position_embeddings: int = 64hidden_act: str = 'gelu_pytorch_tanh'layer_norm_eps: float = 1e-06attention_dropout: float | int = 0.0pad_token_id: int | None = 1bos_token_id: int | None = 49406eos_token_id: int | list[int] | None = 49407projection_size: int | None = None ) Parameters vocab_size (int, optional, defaults to 32000) — Vocabulary size of the model. Defines the number of different tokens that can be represented by the input_ids. hidden_size (int, optional, defaults to 768) — Dimension of the hidden representations. intermediate_size (int, optional, defaults to 3072) — Dimension of the MLP representations. num_hidden_layers (int, optional, defaults to 12) — Number of hidden layers in the Transformer decoder. num_attention_heads (int, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer decoder. max_position_embeddings (int, optional, defaults to 64) — The maximum sequence length that this model might ever be used with. hidden_act (str, optional, defaults to gelu_pytorch_tanh) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc. layer_norm_eps (float, optional, defaults to 1e-06) — The epsilon used by the layer normalization layers. attention_dropout (Union[float, int], optional, defaults to 0.0) — The dropout ratio for the attention probabilities. pad_token_id (int, optional, defaults to 1) — Token id used for padding in the vocabulary. bos_token_id (int, optional, defaults to 49406) — Token id used for beginning-of-stream in the vocabulary. eos_token_id (Union[int, list[int]], optional, defaults to 49407) — Token id used for end-of-stream in the vocabulary. projection_size (int, optional) — Dimensionality of text and vision projection layers. This is the configuration class to store the configuration of a SiglipModel. It is used to instantiate a Siglip model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the google/siglip-base-patch16-224 Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information. Example: Copied >>> from transformers import SiglipTextConfig, SiglipTextModel >>> # Initializing a SiglipTextConfig with google/siglip-base-patch16-224 style configuration >>> configuration = SiglipTextConfig() >>> # Initializing a SiglipTextModel (with random weights) from the google/siglip-base-patch16-224 style configuration >>> model = SiglipTextModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config SiglipVisionConfig class transformers.SiglipVisionConfig < source > ( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: typing.Union[str, ForwardRef('torch.dtype'), NoneType] = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: typing.Optional[typing.Literal['regression', 'single_label_classification', 'multi_label_classification']] = Nonehidden_size: int = 768intermediate_size: int = 3072num_hidden_layers: int = 12num_attention_heads: int = 12num_channels: int = 3image_size: int | list[int] | tuple[int, int] = 224patch_size: int | list[int] | tuple[int, int] = 16hidden_act: str = 'gelu_pytorch_tanh'layer_norm_eps: float = 1e-06attention_dropout: float | int = 0.0 ) Parameters hidden_size (int, optional, defaults to 768) —

SigLIP 2 object embeddings # Role in VSS # The RT-CV microservice ( Object Detection and Tracking ) can produce object embeddings from detection crops and text embeddings from prompts using a combined CLIP-style ONNX export: vision on TensorRT, text on ONNX Runtime with tokenizer assets. Runtime INI keys and APIs are documented under ReID and Embeddings on that page. Note Purpose-built model cards (inputs, outputs, licensing, integration notes) are maintained alongside NGC collaterals. For SigLIP 2, see SigLIP_2 under ngc-collaterals/multimodal/docs/purpose_built_models/ in the collaterals repo. Use those files together with the published TAO embedding guides when you need field-level model card detail. SigLIP 2 at a glance # SigLIP 2 is a vision-language encoder that produces joint image and text embeddings for cross-modal retrieval, object search, and re-identification. The VSS 3.2.0 models section carries SigLIP 2 for RT-CV object embeddings. Variants include SO400M and giant backbones with image resolutions such as 224x224, 256x256, 384x384, and 512x512, and embedding dimensions from 768 to 1536 depending on the checkpoint. Typical TAO entry uses Hugging Face weights; model.type selects the variant (for example siglip2-so400m-patch16-256 or siglip2-giant-opt-patch16-384). See the SigLIP 2 model card and the SigLIP 2 paper. Hardware and software requirements # Follow the TAO Toolkit Quick Start — Requirements. Use a Linux environment with a supported NVIDIA GPU stack; CLIP training and export use the same TAO Launcher pattern as other embedding models. Dataset requirements # TAO CLIP fine-tuning expects image–text pairs. Two dataset layouts are supported in depth in the TAO docs: Custom directory layout — Images and captions in parallel trees; each.txt caption file shares a basename with its image (images/foo.jpg ↔ captions/foo.txt). Supported image extensions include.jpg,.jpeg,.png. WebDataset (WDS) — Sharded.tar archives with paired image and caption keys; suitable for large or cloud-hosted corpora (including S3 URLs via shard lists). For inference-time text embedding, provide a plain-text file with one prompt per line and set inference.text_file in the experiment spec. Full layout examples and YAML snippets are in the CLIP introduction (mirrors the in-repo tlt-docs/docs/text/embedding/clip/overview.rst). Supported model.type values (summary) # The TAO CLIP introduction documents all CLIP-family options. The table below is the VSS 3.2.0 reference for RT-CV object embeddings. Family Example model.type values Typical image size Embedding dim Weights source SigLIP 2 siglip2-so400m-patch16-256, siglip2-so400m-patch16-384, siglip2-giant-opt-patch16-384, fixed-resolution variants listed on NGC 224-512 768-1536 Hugging Face / NGC Deployment limitations (read before export) # TensorRT deployment supports FP16 and FP32 for generated engines. The siglip2-so400m-patch16-naflex variant supports dynamic resolution in training but cannot be exported to ONNX or TensorRT in current TAO—choose a fixed-resolution SigLIP 2 variant for RT-CV deployment. attention_mask may appear as an ONNX input for backward compatibility; behavior is documented in the CLIP introduction. TAO fine-tuning configuration # Fine-tuning uses a YAML experiment specification file (same pattern as other TAO embedding models). The spec is organized into sections that mirror the CLIP training documentation and the in-repo sources under tlt-docs/docs/text/embedding/clip/training.rst. Invoke tasks with: tao model clip <action> -e /path/to/experiment_spec.yaml [overrides] # TensorRT engine generation uses TAO Deploy: tao deploy clip gen_trt_engine -e /path/to/experiment_spec.yaml Supported actions: train, evaluate, inference, export, and (TAO Deploy) gen_trt_engine. An experiment specification file consists of several main components: model — backbone type, optional adaptor_name, freeze flags, text canonicalization train — epochs, GPUs, loss (siglip / clip), precision, optimizer (train.optim), resume paths dataset — train / val loaders (custom or wds), augmentation evaluate — checkpoint and batch size for retrieval metrics inference — image dirs, text_file for prompts export — ONNX path, encoder_type (combined or separate), input height/width, opset gen_trt_engine — ONNX input, TRT output path, TensorRT precision and batch bounds For field-by-field tables (defaults and valid options), use the published CLIP training page (sourced from tlt-docs). Example experiment specification # The following example follows the TAO Launcher layout for SigLIP 2 (siglip2-so400m-patch16-256): custom image-caption dataset, combined ONNX export at 256x256, FP16 TensorRT profile. Adjust model.type and export resolution if you use another fixed-resolution SigLIP 2 checkpoint. results_dir: /results/clip_experiment model: type: siglip2-so400m-patch16-256 adaptor_name: null freeze_vision_encoder: false freeze_text_encoder: false canonicalize_text: false train: num_epochs: 100 num_gpus: 1 num_nodes: 1 checkpoint_interval: 10 resume_training_checkpoint_path: null pretrained_model_path: null loss_type: siglip precision: fp16 grad_checkpointing: false grad_clip_norm: null distributed_strategy: ddp validation_interval: 1 val_check_interval: null optim: optimizer_type: adamw vision_lr: 1.0e-4 text_lr: 1.0e-4 weight_decay: 1.0e-4 betas: [0.9, 0.95] eps: 1.0e-6 warmup_steps: 100 scheduler: cosine dataset: seed: 42 train: type: custom datasets: - image_dir: /data/train/images caption_dir: /data/train/captions caption_file_suffix:.txt image_list_file: null batch_size: 16 num_workers: 8 val: datasets: - image_dir: /data/val/images caption_dir: /data/val/captions batch_size: 16 num_workers: 8 augmentation: scale: [0.4, 1.0] color_jitter: [0.8, 0.32, 0.32, 0.32, 0.08] grayscale: 0.2 evaluate: checkpoint: /results/clip_experiment/train/epoch_100.pth batch_size: 16 inference: checkpoint: /results/clip_experiment/train/epoch_100.pth datasets: - image_dir: /data/inference/images text_file: /data/inference/prompts.txt batch_size: 16 export: checkpoint: /results/clip_experiment/train/epoch_100.pth onnx_file: /results/clip_experiment/export/clip_model.onnx encoder_type: combined input_height: 256 input_width: 256 batch_size: -1 opset_version: 17 gen_trt_engine: onnx_file: /results/clip_experiment/export/clip_model.onnx trt_engine: /results/clip_experiment/deploy/clip_model.engine batch_size: -1 tensorrt: workspace_size: 4096 data_type: fp16 min_batch_size: 1 opt_batch_size: 8 max_batch_size: 16 Launch model fine-tuning # Below are important fields to set before launching training (match paths to your dataset): dataset: train: datasets: - image_dir:?? # training images caption_dir:?? # parallel captions (.txt per image) caption_file_suffix:.txt image_list_file: null val: datasets: - image_dir:?? caption_dir:?? model: type: siglip2-so400m-patch16-256 adaptor_name: null train: num_epochs:?? # e.g. 50–100 for domain fine-tune num_gpus:?? # or pass train.num_gpus=4 on the CLI pretrained_model_path: null # or path to a TAO checkpoint loss_type: siglip precision: fp16 Launch training: tao model clip train -e /path/to/experiment_spec.yaml \ results_dir=/results/clip_run1 \ train.num_gpus=4 Evaluate the fine-tuned model # Point evaluate.checkpoint at your trained.pth (or leave unset for zero-shot pretrained evaluation per TAO docs). tao model clip evaluate -e /path/to/experiment_spec.yaml \ evaluate.checkpoint=/results/clip_experiment/train/epoch_100.pth Run inference (embedding extraction) # Inference writes HDF5 outputs under results_dir (image and text embeddings). Configure inference.datasets and inference.text_file as in the example spec above. tao model clip inference -e /path/to/experiment_spec.yaml \ inference.checkpoint=/results/clip_experiment/train/epoch_100.pth \ inference.text_file=/data/prompts.txt \ results_dir=/results/clip_experiment/inference Export the fine-tuned model # Use export.encoder_type: combined for a single ONNX for RT-CV-style deployment (vision + text). Match export.input_height / input_width to the trained resolution. Export also emits _config.yaml and a tokenizer directory beside the ONNX—keep them together for TAO Deploy. tao model clip export -e /path/to/experiment_spec.yaml \ export.checkpoint=/results/clip_experiment/train/epoch_100.pth \ export.onnx_file=/results/clip_experiment/export/clip_model.onnx \ export.encoder_type=combined Generate a TensorRT engine (TAO Deploy) # tao deploy clip gen_trt_engine -e /path/to/experiment_spec.yaml \ gen_trt_engine.onnx_file=/results/clip_experiment/export/clip_model.onnx \ gen_trt_engine.trt_engine=/results/clip_experiment/deploy/clip_model.engine \ gen_trt_engine.tensorrt.data_type=fp16 Further export shapes, separate vision/text ONNX, tokenizer artifacts, and ONNX external-data notes are in the Exporting the Model section of the CLIP training documentation. Integrating fine-tuned weights into RT-CV # Export a combined image+text ONNX (and tokenizer directory) so vision and text paths stay aligned. Deploy artifacts beside the RT-CV container or mount them with volumes. In DeepStream, enable the vision encoder and text embedder blocks; set onnx-model-path and tokenizer-dir; rebuild the TensorRT engine for the vision branch. Keep image size, tokenizer max length, and normalization consistent with training. Validate REST text embedding APIs, Kafka embedding dimensions, and downstream services (for example Behavior Analytics ). Note Changing embedding dimension or geometry usually requires re-indexing vectors and rechecking similarity thresholds—see Model customization overview. Source material in this repo’s doc trees # The following paths (local checkouts) align with the published TAO URLs above and are useful when drafting or reviewing details: tlt-docs/docs/text/embedding/ — index.rst, clip/overview.rst, clip/training.rst, clip/applications.rst ngc-collaterals/multimodal/docs/purpose_built_models/ — SigLIP_2/overview.md On this page