We address the following generic wave problem: is the estimation of an arbitrarily complex linear $N$-port system's scattering matrix possible if waves can be input and output only via $N_\mathrm{A}<N$ ports while the remaining $N_\mathrm{S}=N-N_\mathrm{A}$ ports are terminated with tunable loads? Fundamentally, this problem is intriguing because it ultimately probes to what extent inherent structure in Maxwell's equations constrains the scattering coefficients. Various limited versions of the problem are of temporary scientific and technological interest, ranging from optimal non-invasive focusing on perturbation-inducing targets in complex media, via the characterization of miniaturized, embedded, receive-only and/or multi-element antenna systems to physics-compliant end-to-end channel models for complex metasurface-programmable "smart radio environments". More generally, solutions to the problem may yield promising measurement techniques to characterize an arbitrary linear $N$-port system with an $N_\mathrm{A}$-port measurement device, where $N_\mathrm{A} \ll N$. We show theoretically that if $N_\mathrm{A}\geq 2$ and at least three distinct tunable loads are available, the problem can be solved except for sign ambiguities on the off-diagonal scattering coefficients involving the $N_\mathrm{S}$ not-directly-accessible (NDA) ports. If the transmission from at least one accessible port to the NDA ports can be measured, the sign ambiguity can be lifted. We corroborate our results with microwave experiments on an 8-port chaotic cavity with $N_\mathrm{A}=N_\mathrm{S}=4$. Moreover, we reveal additional constraining structure in Maxwell's equations by showing that a limitation to phase-insensitive measurements only results in a mild additional blockwise phase ambiguity that can be lifted simultaneously with the sign ambiguity.