We demonstrate that there exist adversarial examples which are just “bugs”:
aberrations in the classifier that are not intrinsic properties of the data distribution.
In particular, we give a new method for constructing adversarial examples which:
- Do not transfer between models, and
-
Do not leak “non-robust features” which allow for learning, in the
sense of Ilyas-Santurkar-Tsipras-Engstrom-Tran-Madry
.
We replicate the Ilyas et al.
experiment of training on mislabeled adversarially-perturbed images
(Section 3.2 of
and show that it fails for our construction of adversarial perturbations.
The message is, whether adversarial examples are features or bugs depends
on how you find them — standard PGD finds features, but bugs are abundant as well.
We also give a toy example of a data distribution which has no “non-robust features”
(under any reasonable definition of feature), but for which standard training yields a highly non-robust
classifier.
This demonstrates, again, that adversarial examples can occur even if the data distribution does not
intrinsically have any vulnerable directions.
Background
Many have understood Ilyas et al.
to claim that adversarial examples are not “bugs”, but are “features”.
Specifically, Ilyas et al. postulate the following two worlds:
- World 1: Adversarial examples exploit directions irrelevant for classification (“bugs”).
In this world, adversarial examples occur because classifiers behave
poorly off-distribution,
when they are evaluated on inputs that are not natural images.
Here, adversarial examples would occur in arbitrary directions,
having nothing to do with the true data distribution. -
World 2: Adversarial examples exploit useful directions for classification (“features”).
In this world, adversarial examples occur in directions that are still “on-distribution”,
and which contain features of the target class.
For example, consider the perturbation that
makes an image of a dog to be classified as a cat.
In World 2, this perturbation is not purely random, but has something to do with cats.
Moreover, we expect that this perturbation transfers to other classifiers trained to distinguish cats
vs. dogs.
Our main contribution is demonstrating that these worlds are not mutually exclusive — and in fact, we
are in both.
Ilyas et al.
show that there exist adversarial examples in World 2, and we show there exist
examples in World 1.
Constructing Non-transferrable Targeted Adversarial Examples
We propose a method to construct targeted adversarial examples for a given classifier
$f$,
which do not transfer to other classifiers trained for the same problem.
Recall that for a classifier $f$, an input example $(x, y)$, and target class $y_{targ}$,
a targeted adversarial example is an $x’$ such that $||x – x’||\leq \eps$ and
$f(x’) = y_{targ}$.
The standard method of constructing adversarial examples is via Projected Gradient Descent (PGD)
PGD is described in the appendix.
which starts at input $x$, and iteratively takes steps $\{x_t\}$
to minimize the loss $L(f, x_t, y_{targ})$.
That is, we take steps in the direction
$$-\nabla_x L(f, x_t, y_{targ})$$
where $L(f, x, y)$ is the loss of $f$ on input $x$, label $y$.
Note that since PGD steps in the gradient direction towards the target class,
we may expect these adversarial examples have feature leakage from the target class.
For example, suppose we are perturbing an image of a dog into a plane (which usually appears against a blue
background).
It is plausible that the gradient direction tends to make the dog image more blue,
since the “blue” direction is correlated with the plane class.
In our construction below, we attempt to eliminate such feature leakage.
The gradient of the loss can be thought of as having an on-manifold “feature component”
and an off-manifold “random component”.
PGD steps along both components, hence causing feature-leakage in adversarial examples.
Our construction below attempts to step only in the off-manifold direction.
Our Construction
Let $\{f_i : \R^n \to \cY\}_i$ be an ensemble of classifiers
for the same classification problem as $f$.
For example, we can let $\{f_i\}$ be a collection of ResNet18s trained from
different random initializations.
For input example $(x, y)$ and target class $y_{targ}$,
we perform iterative updates to find adversarial attacks — as in PGD.
However, instead of stepping directly in the gradient direction, we
step in the direction
Formally, we replace the iterative step with
$$x_{t+1} \gets \Pi_\eps\left( x_t
– \alpha( \nabla_x L(f, x_t, y_{targ}) + \E_i[ \nabla_x L(f_i, x_t, y)]) \right)$$
where $\Pi_\eps$ is the projection onto the $\eps$-ball around $x$.
$$-\left( \nabla_x L(f, x_t, y_{targ}) + \E_i[ \nabla_x L(f_i, x_t, y)] \right)$$
That is, instead of taking gradient steps to minimize $L(f, x, y_{targ})$,
we minimize the “disentangled loss”
We could also consider explicitly using the ensemble to decorrelate,
by stepping in direction
$\nabla_x L(f, x, y_{targ}) – \E_i[ \nabla_x L(f_i, x, y_{targ})]$.
This works well for small $\epsilon$,
but the given loss has better optimization properties for larger $\epsilon$.
$$L(f, x, y_{targ}) + \E_i[L(f_i, x, y)]$$
This loss encourages finding an $x_t$ which is adversarial for $f$,
but not for the ensemble $\{f_i\}$.
These adversarial examples will not be adversarial for the ensemble $\{f_i\}$. But perhaps surprisingly,
these examples are also not adversarial for
new classifiers trained for the same problem.
Experiments
We train a ResNet18 on CIFAR10 as our target classifier $f$.
For our ensemble, we train 10 ResNet18s on CIFAR10, from fresh random initializations.
We then test the probability that
a targeted attack for $f$
transfers to a new (freshly-trained) ResNet18, with the same targeted class.
Our construction yields adversarial examples which do not transfer well to new models.
For $L_{\infty}$ attacks:
| Attack Success | Transfer Success | |
|---|---|---|
| PGD | 99.6% | 52.1% |
| Ours | 98.6% | 0.8% |
For $L_2$ attacks:
| Attack Success | Transfer Success | |
|---|---|---|
| PGD | 99.9% | 82.5% |
| Ours | 99.3% | 1.7% |
Adversarial Examples With No Features
Using the above, we can construct adversarial examples
which do not suffice for learning.
Here, we replicate the Ilyas et al. experiment
that “Non-robust features suffice for standard classification”
(Section 3.2 of
but show that it fails for our construction of adversarial examples.
To review, the Ilyas et al. non-robust experiment was:
- Train a standard classifier $f$ for CIFAR.
-
From the CIFAR10 training set $S = \{(X_i, Y_i)\}$,
construct an alternate train set $S’ = \{(X_i^{Y_i \to (Y_i + 1)}, Y_i + 1)\}$,
where $X_i^{Y_i \to (Y_i +1)}$ denotes an adversarial example for
$f$, perturbing $X_i$ from its true class $Y_i$ towards target class $Y_i+1 (\text{mod }10)$.
Note that $S’$ appears to humans as “mislabeled examples”. -
Train a new classifier $f’$ on train set $S’$.
Observe that this classifier has non-trivial accuracy on the original CIFAR distribution.
Ilyas et al. use Step (3) to argue that
adversarial examples have a meaningful “feature” component.
However, for adversarial examples constructed using our method, Step (3) fails.
In fact, $f’$ has good accuracy with respect to the “label-shifted” distribution
$(X, Y+1)$, which is intuitively what we trained on.
For $L_{\infty}$ attacks:
| Test Acc on CIFAR: $(X, Y)$ | Test Acc on Shifted-CIFAR: $(X, Y+1)$ | |
|---|---|---|
| PGD | 23.7% | 40.4% |
| Ours | 2.5% | 75.9% |
Table: Test Accuracies of $f’$
For $L_2$ attacks:
| Test Acc on CIFAR: $(X, Y)$ | Test Acc on Shifted-CIFAR: $(X, Y+1)$ | |
|---|---|---|
| PGD | 33.2% | 27.3% |
| Ours | 2.8% | 70.8% |
Table: Test Accuracies of $f’$
Adversarial Squares: Adversarial Examples from Robust Features
To further illustrate that adversarial examples can be “just bugs”,
we show that they can arise even when the true data distribution has no “non-robust features” — that is, no intrinsically vulnerable directions.
We are unaware of a satisfactory definition of “non-robust feature”, but we claim that for any
reasonable
intrinsic definition, this problem has no non-robust features.
Intrinsic here meaning, a definition which depends only on geometric properties of the data
distribution, and not on the family of classifiers, or the finite-sample training set.
We do not use the Ilyas et al. definition of “non-robust features,” because we believe it is vacuous.
In particular, by the Ilyas et al. definition, every distribution
has “non-robust features” — so the definition does not discern structural properties of the
distribution.
Moreover, for every “robust feature” $f$, there exists a corresponding “non-robust feature” $f’$, such
that $f$ and $f’$ agree on the data distribution — so the definition depends strongly on the
family of classifiers being considered.
In the following toy problem, adversarial vulnerability arises as a consequence of finite-sample
overfitting, and
label noise.
The problem is to distinguish between CIFAR-sized images that are either all-black or all-white,
with a small amount of random pixel noise and label noise.
A sample of images from the distribution.
Formally, let the distribution be as follows.
Pick label $Y \in \{\pm 1\}$ uniformly,
and let $$X :=
\begin{cases}
(+\vec{\mathbb{1}} + \vec\eta_\eps) \cdot \eta & \text{if $Y=1$}\\
(-\vec{\mathbb{1}} + \vec\eta_\eps) \cdot \eta & \text{if $Y=-1$}\\
\end{cases}$$
where $\vec\eta_\eps \sim [-0.1, +0.1]^d$ is uniform $L_\infty$ pixel noise,
and
$\eta \in \{\pm 1\} \sim Bernoulli(0.1)$ is the 10% label noise.
A plot of samples from a 2D-version of this distribution is shown to the right.
Notice that there exists a robust linear classifier for this problem which achieves perfect robust
classification, with up to $\eps = 0.9$ magnitude $L_\infty$ attacks.
However, if we sample 10000 training images from this distribution, and train
a ResNet18 to 99.9% train accuracy,
We optimize using Adam with learning-rate $0.00001$ and batch size $128$ for 20 epochs.
the resulting classifier is highly non-robust:
an $\eps=0.01$ perturbation suffices to flip the class of almost all test examples.
The input-noise and label noise are both essential for this construction.
One intuition for what is happening is: in the initial stage of training
the optimization learns the “correct” decision boundary (indeed, stopping after 1 epoch results in a robust
classifier).
However, optimizing for close to 0 train-error requires a network with high Lipshitz constant
to fit the label-noise, which hurts robustness.
Left: The training set (labels color-coded). Middle: The classifier after 10 SGD steps.
Right: The classifier at the end of training. Note that it is overfit, and not robust.
Figure adapted from
Addendum: Data Poisoning via Adversarial Examples
As an addendum, we observe that the “non-robust features”
experiment of
directly implies data-poisoning attacks:
An adversary that is allowed to imperceptibly change every image in the training set can destroy the
accuracy of the learnt classifier — and can moreover apply an arbitrary permutation
to the classifier output labels (e.g. swapping cats and dogs).
To see this, recall that the original “non-robust features” experiment shows:
notation, and also using vanilla PGD to find adversarial examples.
1. If we train on distribution $(X^{Y \to (Y+1)}, Y+ 1)$ the classifier learns to predict well
on distribution $(X, Y)$.
By permutation-symmetry of the labels, this implies that:
2. If we train on distribution $(X^{Y \to (Y+1)}, Y)$ the classifier learns to predict well
on distribution $(X, Y-1)$.
Note that in case (2), we are training with correct labels, just perturbing the inputs imperceptibly,
but the classifier learns to predict the cyclically-shifted labels.
Concretely, using the original numbers of
Table 1 in
an adversary can perturb the CIFAR10 train set by $\eps=0.5$ in $L_2$,
and cause the learnt classifier to output shifted-labels
43.7% of the time
(cats classified as birds, dogs as deers, etc).
This should extend to attacks that force arbitrary desired permutations of the labels.
Leave a Reply