Here, we present a new approach based on manifold learning for knowledge discovery and inverse design with minimal complexity in photonic nanostructures. Our approach builds on studying sub-manifolds of responses of a class of nanostructures with different design complexities in the latent space to obtain valuable insight about the physics of device operation to guide a more intelligent design. In contrast to the current methods for inverse design of photonic nanostructures, which are limited to pre-selected and usually over-complex structures, we show that our method allows evolution from an initial design towards the simplest structure while solving the inverse problem.