We investigate the construction of generative models capable of encoding physical constraints that can be hard to express explicitly. For the problem of inverse material design, where one seeks to design a material with a prescribed set of properties, a significant challenge is ensuring synthetic viability of a proposed new material. We encode an implicit dataset relationships, namely that certain materials can be decomposed into other ones in the dataset, and present a VAE model capable of preserving this property in the latent space and generating new samples with the same. This is particularly useful in sequential inverse material design, an emergent research area that seeks to design a material with specific properties by sequentially adding (or removing) elements using policies trained through deep reinforcement learning.