Harsh Pal

Why CLIP Changed Everything for Vision-Language Models

• 10 min read

The Problem Before CLIP

Traditional image classifiers are trained on a fixed set of categories. An ImageNet model learns to distinguish 1,000 classes, and that is all it can ever predict. If you need a new class — say, "a photo of a traffic cone" — you have to collect labelled examples and retrain. This is expensive, slow, and does not scale.

The fundamental limitation is that vision models and language models lived in separate worlds. Images were mapped to class indices, not to the rich semantic space that language provides. CLIP changed that.

What is CLIP?

CLIP (Contrastive Language–Image Pre-training), introduced by Radford et al. (2021) at OpenAI, learns to connect images and text by training on 400 million image–text pairs scraped from the internet. Instead of predicting a fixed class, CLIP learns a shared embedding space where matching images and captions are close together.

CLIP has two encoders:

  • Image encoder $f_I$: Takes an image $x$ and produces a vector $\mathbf{v} = f_I(x) \in \mathbb{R}^d$. This can be a ResNet or a Vision Transformer (ViT).
  • Text encoder $f_T$: Takes a text string $t$ and produces a vector $\mathbf{u} = f_T(t) \in \mathbb{R}^d$. This is a Transformer.

Both vectors live in the same $d$-dimensional space. The similarity between an image and a text is simply the cosine similarity:

$$\text{sim}(x, t) = \frac{\mathbf{v} \cdot \mathbf{u}}{\|\mathbf{v}\| \, \|\mathbf{u}\|}$$

Contrastive Training Objective

CLIP is trained with a symmetric contrastive loss. Given a batch of $N$ image–text pairs $\{(x_i, t_i)\}_{i=1}^N$, we compute the $N \times N$ matrix of cosine similarities. The diagonal entries are the matching pairs; everything off-diagonal is a negative.

For each image $x_i$, the probability that $t_j$ is the correct caption is:

$$p_{i \to j} = \frac{\exp(\text{sim}(x_i, t_j) / \tau)}{\sum_{k=1}^{N} \exp(\text{sim}(x_i, t_k) / \tau)}$$

where $\tau$ is a learnable temperature parameter. The image-to-text loss for sample $i$ is the cross-entropy:

$$\ell_i^{I \to T} = -\log \, p_{i \to i}$$

An analogous text-to-image loss $\ell_i^{T \to I}$ is computed by going the other direction. The total loss averages both:

$$\mathcal{L} = \frac{1}{2N} \sum_{i=1}^{N} \left( \ell_i^{I \to T} + \ell_i^{T \to I} \right)$$

This objective pushes matching pairs together and non-matching pairs apart in the shared embedding space. With large batch sizes (CLIP used 32,768), there are plenty of negatives per sample, which helps learn fine-grained distinctions.

Zero-Shot Classification

The key insight of CLIP is that classification becomes a retrieval problem. To classify an image into one of $C$ classes, you:

  1. Create text prompts for each class, e.g., "a photo of a {class name}".
  2. Encode all prompts: $\mathbf{u}_c = f_T(\text{"a photo of a } c \text{"})$ for $c = 1, \ldots, C$.
  3. Encode the image: $\mathbf{v} = f_I(x)$.
  4. Predict the class with highest similarity: $$\hat{y} = \arg\max_{c} \; \text{sim}(\mathbf{v}, \mathbf{u}_c)$$

No training on the target dataset is needed. CLIP achieves competitive accuracy with fully-supervised baselines on many benchmarks purely through this zero-shot approach. On ImageNet, zero-shot CLIP (ViT-L/14) matches the accuracy of a supervised ResNet-50.

Why Does This Matter?

CLIP unlocked several capabilities that were not practical before:

  • Open-vocabulary recognition: You can classify images into arbitrary categories just by changing the text prompt — no retraining needed.
  • Robust representations: CLIP features are much more robust to distribution shift than ImageNet-trained features, because they were trained on diverse, naturally-occurring internet data.
  • Foundation for downstream tasks: CLIP's embedding space is the backbone for text-to-image generation (DALL-E, Stable Diffusion), visual question answering, image retrieval, and more.
  • Prompt engineering for vision: Just as GPT responds to different text prompts, CLIP's predictions change with the prompt template. "A photo of a dog" vs. "A sketch of a dog" can yield different behaviour — opening up a new way to control vision models.

Prompt Learning with CLIP

Manually writing text prompts is limited. CoOp (Zhou et al., 2022) replaces the hand-crafted prompt with learnable context vectors. Instead of "a photo of a [class]", the prompt becomes:

$$\mathbf{u}_c = f_T([\mathbf{v}_1][\mathbf{v}_2] \cdots [\mathbf{v}_M][\text{class}_c])$$

where $\mathbf{v}_1, \ldots, \mathbf{v}_M$ are learnable token embeddings optimized on a few labelled examples from the target dataset. Only these vectors are updated — CLIP's encoders stay frozen. This is far more parameter-efficient than fine-tuning the entire model and often outperforms hand-crafted prompts by a significant margin.

CoCoOp (Zhou et al., 2022) further extends this by conditioning the prompt on the input image, making the learned prompts generalizable across different visual domains.

Limitations

CLIP is not perfect:

  • Fine-grained tasks: It struggles with tasks requiring detailed spatial reasoning, like counting objects or understanding relative positions.
  • Compositionality: "A red car on a blue road" vs. "A blue car on a red road" can confuse CLIP, because the bag-of-words nature of contrastive training does not always capture attribute bindings.
  • Bias: Trained on internet data, CLIP inherits societal biases present in that data.

References

  1. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever. "Learning Transferable Visual Models From Natural Language Supervision." ICML, 2021.
  2. K. Zhou, J. Yang, C. C. Loy, Z. Liu. "Learning to Prompt for Vision-Language Models (CoOp)." IJCV, 130(9):2337–2348, 2022.
  3. K. Zhou, J. Yang, C. C. Loy, Z. Liu. "Conditional Prompt Learning for Vision-Language Models (CoCoOp)." CVPR, 2022.
  4. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT)." ICLR, 2021.
  5. T. Chen, S. Kornblith, M. Norouzi, G. Hinton. "A Simple Framework for Contrastive Learning of Visual Representations (SimCLR)." ICML, 2020.