Harsh Pal

Understanding Domain Adaptation

• 8 min read

What is Domain Adaptation?

Machine learning models assume that training data and test data come from the same distribution. In practice, this assumption rarely holds. A model trained to classify product reviews from Amazon may perform poorly on Yelp restaurant reviews, even though both tasks involve sentiment analysis. The data distributions differ — this gap is called domain shift.

Domain adaptation is a family of techniques that help a model trained on a source domain perform well on a different but related target domain. Formally, we have:

  • Source domain $\mathcal{D}_S = \{(x_i^s, y_i^s)\}_{i=1}^{n_s}$ — labelled data we can train on.
  • Target domain $\mathcal{D}_T = \{x_j^t\}_{j=1}^{n_t}$ — data from a different distribution, often with few or no labels.

The goal is to learn a model $f$ that minimizes the target risk $\epsilon_T(f) = \mathbb{E}_{(x,y) \sim \mathcal{D}_T}[\ell(f(x), y)]$, even though we primarily have access to labelled data from $\mathcal{D}_S$.

Why Does Domain Shift Happen?

Domain shift arises because the joint distribution $P(X, Y)$ changes between domains. This can manifest in several ways:

  • Covariate shift: The input distribution changes, $P_S(X) \neq P_T(X)$, but the labelling function stays the same, $P_S(Y|X) = P_T(Y|X)$. Example: training on studio photos, testing on outdoor photos.
  • Label shift: The class proportions change, $P_S(Y) \neq P_T(Y)$, but the class-conditional input distribution stays the same. Example: a disease becomes more prevalent in a new population.
  • Concept shift: The relationship between inputs and labels changes, $P_S(Y|X) \neq P_T(Y|X)$. Example: the word "sick" meaning "ill" vs. slang for "cool".

The Theory Behind It

Ben-David et al. (2010) provided a foundational bound on target error. For a hypothesis $h$, the target error $\epsilon_T(h)$ is bounded by:

$$\epsilon_T(h) \leq \epsilon_S(h) + d_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{D}_S, \mathcal{D}_T) + \lambda^*$$

This bound tells us three things:

  1. Source error $\epsilon_S(h)$: The model should perform well on the source domain.
  2. Domain divergence $d_{\mathcal{H}\Delta\mathcal{H}}$: The distance between the two distributions should be small. This is the term we can actively minimize.
  3. Ideal joint error $\lambda^*$: The error of the ideal hypothesis that works well on both domains. This is a constant — if the two tasks are fundamentally incompatible, no adaptation method will help.

Most domain adaptation methods focus on minimizing the second term — making the feature representations of the source and target domains indistinguishable.

Core Approaches

1. Discrepancy-Based Methods

These methods explicitly measure and minimize the distance between source and target feature distributions. A popular measure is Maximum Mean Discrepancy (MMD):

$$\text{MMD}(\mathcal{D}_S, \mathcal{D}_T) = \left\| \frac{1}{n_s}\sum_{i=1}^{n_s}\phi(x_i^s) - \frac{1}{n_t}\sum_{j=1}^{n_t}\phi(x_j^t) \right\|_{\mathcal{H}}$$

where $\phi(\cdot)$ maps samples into a Reproducing Kernel Hilbert Space (RKHS). Intuitively, MMD compares the mean embeddings of two distributions — if they are close, the distributions are similar. Deep Adaptation Networks (Long et al., 2015) add MMD penalties to the loss function of a deep network, encouraging the learned features to have similar distributions across domains.

2. Adversarial Methods

Inspired by GANs, adversarial domain adaptation uses a domain discriminator $D$ that tries to distinguish source features from target features, while the feature extractor $G$ tries to fool it. The minimax objective is:

$$\min_G \max_D \; \mathbb{E}_{x \sim \mathcal{D}_S}[\log D(G(x))] + \mathbb{E}_{x \sim \mathcal{D}_T}[\log(1 - D(G(x)))]$$

When the discriminator can no longer tell the domains apart, the features are domain-invariant. Domain-Adversarial Neural Networks (DANN) by Ganin et al. (2016) implement this elegantly using a gradient reversal layer — during backpropagation, gradients from the discriminator are negated before reaching the feature extractor, which makes standard SGD perform the minimax optimization.

3. Self-Training / Pseudo-Labelling

A simpler but effective approach: train on source data, generate pseudo-labels for confident target predictions, then retrain on both. The key is selecting reliable pseudo-labels. Common strategies include confidence thresholding — keeping only predictions where $\max_c \; P(y = c \mid x) > \tau$ for a threshold $\tau$ — and curriculum-style methods that gradually include harder examples.

Variants of Domain Adaptation

  • Unsupervised DA: No target labels available. This is the most studied and challenging setting.
  • Semi-supervised DA: A few target labels are available alongside abundant unlabelled target data.
  • Multi-source DA: Multiple source domains are available, e.g., adapting from both Amazon and Twitter reviews to Yelp.
  • Domain generalization: No target data at all during training — the model must generalize to unseen domains purely from diverse source training.

A Simple Example

Consider digit recognition: you have labelled MNIST digits (clean, centred, black-and-white) and want to classify SVHN digits (colourful, noisy, from street photos). A standard CNN trained on MNIST gets roughly 60% accuracy on SVHN. Adding a DANN-style adversarial loss to align the feature distributions can push this to around 75–80%, without using a single SVHN label.

References

  1. S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, J. W. Vaughan. "A theory of learning from different domains." Machine Learning, 79(1–2):151–175, 2010.
  2. M. Long, Y. Cao, J. Wang, M. I. Jordan. "Learning Transferable Features with Deep Adaptation Networks." ICML, 2015.
  3. Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, V. Lempitsky. "Domain-Adversarial Training of Neural Networks." JMLR, 17(1):1–35, 2016.
  4. K. Saito, K. Watanabe, Y. Ushiku, T. Harada. "Maximum Classifier Discrepancy for Unsupervised Domain Adaptation." CVPR, 2018.
  5. D. Lee. "Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks." ICML Workshop, 2013.