Data Driven Testing of Cyber Physical Systems

Consumer grade cyber-physical systems are becoming an integral part of our life, automatizing and simplifying everyday tasks; they are almost always capable of connection to the network allowing remote monitoring and programming. They rely on powerful programming languages, cloud infrastructures, and ultimately on complex software stacks. Indeed, due to complex interactions between hardware, networking and software, developing and testing such systems is known to be a challenging task. Ensuring properties such as dependability, security or data confidentiality is far from obvious. Various quality assurance and testing strategies have been proposed. The most common approach for pre-deployment testing is to model the system and run simulations with models or software in the loop. In practice, most often, tests are run for a small number of simulations, which are selected based on the engineers' domain knowledge and experience. We have implemented our approach in Python, using standard frameworks and used it to generate scenarios violating temperature constraints for a smart thermostat implemented as a part of our IoT testbed. Data collected from an application managing a smart building have been used to learn models of the environment under ever changing conditions. The suggested approach allowed us to identify several pit-fails, scenarios (i.e. environment conditions), where the system behaves not as expected.