Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective

Modern software systems are often highly configurable to tailor varied requirements from diverse stakeholders. Understanding the mapping between configurations and the desired performance attributes plays a fundamental role in advancing the controllability and tuning of the underlying system, yet has long been a dark hole of knowledge due to its black-box nature. While there have been previous efforts in performance analysis for these systems, they analyze the configurations as isolated data points without considering their inherent spatial relationships. This renders them incapable of interrogating many important aspects of the configuration space like local optima. In this work, we advocate a novel perspective to rethink performance analysis -- modeling the configuration space as a structured ``landscape''. To support this proposition, we designed \our, an open-source, graph data mining empowered fitness landscape analysis (FLA) framework. By applying this framework to $86$M benchmarked configurations from $32$ running workloads of $3$ real-world systems, we arrived at $6$ main findings, which together constitute a holistic picture of the landscape topography, with thorough discussions about their implications on both configuration tuning and performance modeling.

LONViZ: Unboxing the black-box of Configurable Software Systems from a Complex Networks Perspective

Most, if not all, modern software systems are highly configurable to tailor both their functional and non-functional properties to a variety of stakeholders. Due to the black-box nature, it is difficult, if not impossible, to analyze and understand its behavior, such as the interaction between combinations of configuration options with regard to the performance, in particular, which is of great importance to advance the controllability of the underlying software system. This paper proposes a tool, dubbed LONViZ, which is the first of its kind, to facilitate the exploratory analysis of black-box configurable software systems. It starts from a systematic sampling over the configuration space of the underlying system. Then LONViZ seeks to construct a structurally stable LON by synthesizing multiple repeats of sampling results. Finally, exploratory analysis can be conducted on the stable LON from both qualitative and quantitative perspectives. In experiments, we choose four widely used real-world configurable software systems to develop benchmark platforms under 42 different running environments. From our empirical study, we find that LONViZ enables both qualitative and quantitative analysis and disclose various interesting hidden patterns and properties of different software systems.