On the Tightness of Semidefinite Relaxations for Certifying Robustness to Adversarial Examples

The robustness of a neural network to adversarial examples can be provably certified by solving a convex relaxation. If the relaxation is loose, however, then the resulting certificate can be too conservative to be practically useful. Recently, a less conservative robustness certificate was proposed, based on a semidefinite programming (SDP) relaxation of the ReLU activation function. In this paper, we give a geometric analysis for the tightness of this relaxation. We show that, for a least-squares restriction of the usual adversarial attack problem, the SDP relaxation is tight over a single hidden layer under reasonable assumptions. The resulting robustness certificate is exact, meaning that it provides a lower-bound on the size of the smallest adversarial perturbation, as well as a globally optimal perturbation that attains the lower-bound. For several hidden layers, the SDP relaxation is not usually tight; we give an explanation using the underlying hyperbolic geometry. We experimentally confirm our theoretical insights using a general-purpose interior-point method and a custom rank-2 Burer-Monteiro algorithm.