Neural Injective Functions for Multisets, Measures and Graphs via a Finite Witness Theorem

Injective multiset functions have a key role in the theoretical study of machine learning on multisets and graphs. Yet, there remains a gap between the provably injective multiset functions considered in theory, which typically rely on polynomial moments, and the multiset functions used in practice which typically rely on $\textit{neural moments}$, whose injectivity on multisets has not been studied to date. In this paper we bridge this gap by showing that moments of neural network do define an injective multiset function, provided that an analytic non-polynomial activation is used. The number of moments required by our theory is optimal up to a multiplicative factor of two. To prove this result, we state and prove a $\textit{finite witness theorem}$, which is of independent interest. As a corollary to our main theorem, we derive new approximation results for functions on multisets and measures, and new separation results for graph neural networks. We also provide two negative results: We show that (1) moments of piecewise-linear neural networks do not lead to injective multiset functions, and (2) even when moment-based multiset functions are injective, they will never be bi-Lipschitz.