Fairness Through Computationally-Bounded Awareness

We study the problem of fair classification within the versatile framework of Dwork et al. [ITCS 2012], which assumes the existence of a metric that measures similarity between pairs of individuals. Unlike previous works on metric-based fairness, we do not assume that the entire metric is known to the learning algorithm. Instead, we study the setting where a learning algorithm can query this metric a bounded number of times to ascertain similarities between particular pairs of individuals. For example, the queries might be answered by a panel of specialists spanning social scientists, statisticians, demographers, and ethicists. We propose "metric multifairness," a new definition of fairness that is parameterized by a similarity metric $\delta$ on pairs of individuals and a collection ${\cal C}$ of"comparison sets" over pairs of individuals. One way to view this collection is as the family of comparisons that can be expressed within some computational bound. With this interpretation, metric multifairnesss loosely guarantees that similar subpopulations are treated similarly, as long as these subpopulations can be identified within this bound. In particular, metric multifairness implies that a rich class of subpopulations are protected from a multitude of discriminatory behaviors. We provide a general-purpose framework for learning a metric multifair hypothesis that achieves near-optimal loss from a small number of random samples from the metric $\delta$. We study the sample complexity and time complexity of learning a metric multifair hypothesis (providing rather tight upper and lower bounds) by connecting it to the task of learning the class ${\cal C}$. In particular, if the class ${\cal C}$ admits an efficient agnostic learner, then we can learn such a metric multifair hypothesis efficiently.